X does have something to do with it in my opinion, it's not integrated right into the system (which is both good and bad) and is a little slow. I would personally like to see fresco get it's feet off the ground, but that's just me. The windows GUI is partly written in assembly to speed things up, which could have something to do with it aswell, although there may be assembly used in X too, I don't know.
If you want to speed up your hard drive, I wrote a tutorial here
My gentoo system is the fastest I've ever used on my computer, and I think it may be largely due to the kernel. I have finally started to really get the hang of compiling my own kernel and find a pretty decent speed improvement. Also, enabling the low latency option in the kernel makes a big difference on the desktop.
As for prelink, there is a tutorial on doing this under gentoo here. I can't see there being too much of a difference for doing this on other distro's once you have prelink installed.