What is the output of the following commands:
It could be a speed / duplex issue with the ethernet NIC.
# netstat -i
# ethtool -S eth0
# mii-tool eth0
if there are lots of errors, make sure the speed and duplex match on the switch and your server. Duplex is best set to "full" and speed to the maximum capable of the NIC which your switch also supports. Sometimes a Linux server will be set to the default auto-negotiate, and the switch coded to full duplex, which forces the NIC into half duplex mode. See this link. The solution is to either set the switch to "auto", or force the NIC to the pre-set values on the switch.
If you are hooked to a hub, and not a switch, it'll always negotiate to half duplex. For best results, a switch is the preferred choice for its full duplex capabilities.
The other problem could be DNS. Sometimes one of the DNS servers for your domain may be incorrectly configured so that sometimes the incorrect IP address is returned for your website. This can cause delays until your browser times out and then decides to try the next DNS server in the list.