I'm not sure. How much memory do you have? I have a 750Mhz P3 with 512MB RAM and it works OK in GUI mode. I don't use it for desktop applications very much, so I wouldn't be much help on the overall performance.
I wouldn't go through the exercise of doing a kernel recompilation unless you feel the system is genuinely slow and hampering your work.
You'll also have to remember that a yum update (like up2date) could erase your efforts unless you exclude the kernel from the update list.
You could do the recompilation as a learning exercise too.
On my really slow systems, 300 MHz Celeron, 128MB RAM, I run in text mode (init level 3). It works OK for simple, automated disk to disk backups of my home server.