if you're sure that your hardware will never change then there is no reason to use kernel modules IMO (except for closed source stuff that is only available as a module like nvidia,vmware,...)
however, suppose you have a hardware problem and you need to replace that part... if you do not have support for that new piece of hardware in your kernel and don't have a modular kernel + the necessary module available on the system (like the one that came with the distro for instance) then you could be in trouble depending on which piece of hardware it is.
i used to build monolithic kernels all the time for my system, but ever since i switched back to distros i just use the kernel that came with the distro. those are always modular and the people who built those kernels generally know a lot more about the kernel than you do. you really won't notice any performance difference with a custom-compiled kernel (unless you need to recompile it to enable a specific feature) and if you do, it's probably the placebo effect. From a security POV it might have some advantages, but you should ask yourself: is it really worth it? i don't think so, but of course you are free to disagree