Gigabit switch

**vwgtiturbo** · 07-11-2006, 02:48 AM

A while ago, I upgraded to all Gigabit NICs and also purchased a Gigabit switch. Now, when I was running Windows (no longer), my file transfers between two Gigabit-enabled systems were SLOW. I mean 10-11MB/s SLOW (as measured through my firewall meter). Needless to say, I was raging mad... spending the cash on the NICs, switch (which is a reknowned LinkSys POS), and the CAT6 cabling. Now, I realize that I will never get the maximum theoretical speed due to overhead and whatnot, but I found it interesting that my speeds were the same as they were with my 10/100 connection. Now, after having the switch for about 4 or 5 months, and using Linux exclusively (and still seeing the slow speeds, measured by rsync's progress meter), I am still seeing slow speeds. However, ONE time, and I mean, ONE time, I did get transfers of 35-48MB/s, which is a lot better than I WAS getting. Alas, this happened only for one backup session, never to be repeated again. Does someone know of a tool that I can use that is a little more reliable, because this is driving me nuts. I backup anywhere from 5-160GB over a LAN every week or so, and this slowness really defeats the purpose of a Gigabit LAN. I checked the status of my NICs and they are set to full duplex, 1000MB link, so I don't know what the hell the problem is... Thanks in advance.

**trickster** · 07-11-2006, 12:27 PM

There seems to be a common misunderstanding that by plugging in a Gigabit Ethernet card, one will magically achieve ten times the performance compared to a machine with a Fast Ethernet connection. To obtain decent performance on a gigabit class network, it is important that both the TCP stack and the application software is properly tuned for the job. If you simply run the same applications which were written and optimised in the Fast Ethernet or even the 10Mbit Ethernet era, do not expect any significant performance improvement.

Squeeze Your Gigabit NIC for Top Performance

Also, your bottleneck might not be the network, but your disk I/O or your system memory or processor.

Another Link

What speed do you get if you bypass the router and connect the servers directly to each other on the gigabit interfaces?

**vwgtiturbo** · 07-13-2006, 03:55 AM

Sorry it took me so long to reply to this... The noisy fan on my LinkSys went out, so instead of filing a claim for warranty service on a noisy, slow-performing switch, I did a little research and bought another (SMC 8505T). Now, here are my informal test results (transferring a DVD ISO of 3.85GB over rsync, which happens to include ssh in the loop by default):

LinkSys: 9-10MB/s solid
SMC: 30-40MB/s until the download hits about 42%. Then the transfer seems to hang for about 3-4 seconds. The download resumes, but this time, only at about 4-5MB/s.

So, the SMC is faster initially, but slows down considerably at a certain point (I did the transfer 3 or 4 times, and it slows down at the same point each time).

I don't have a crossover cable, and honestly, even if I did, I have no idea how I would do that. I have the machines set up with static IPs, so can I just hook up the cable and run with it, or...?

I am going nuts with this. I mean, both machines are fairly beefy (one of them is my sig, the other is basically the same machine, but with an A64 3400+ instead of the 3800+, both are SATA, etc.).

Now, I am not a complete newb, so I knew that I wouldn't get a full gigabit of bandwidth. However, there should be SOME speed increase. Hell, there should be a fairly decent speed increase. If not, how on earth can they sell this crap? False advertising, or what? I don't know... I am just a bit irked...

I am going to read the links you provided and see if I can squeeze anything else out of this... Thanks for the links!

**vwgtiturbo** · 07-13-2006, 04:23 AM

Wow, this really isn't working... I went to this site and performed the Linux tweaks that were supposed to 'skyrocket TCP performance' and my speeds are in the gutter: 3-5.5MB/s, even worse than before, MUCH worse (this is with the SMC that did 30-40MB/s at first, then halfway through, slowed to 5MB/s). I did this to the client and the server to ensure everything was even. The worst part is that I didn't bother to look at what the settings were before I changed them, so now I have no idea what to set them back to... Great...

**trickster** · 07-13-2006, 12:15 PM

If both machines have static ip's and are on the same subnet and segment, you should be ble to connect a creossover Cat5E or CAT 6 cable between machines, and it should work fine.

**Outlaw** · 07-13-2006, 02:10 PM

Don't use file transfers as a measure of network throughput. You will be measuring disk throughput and ssh overhead (if rsync).

here is a PET that may help

**vwgtiturbo** · 07-13-2006, 08:50 PM

I will try to do this testing when I get home (hopefully). It is driving me nuts that it started out SO FAST, then just dies. Then after tweaking, it is even slower than it was originally... The recipient system is a RAID based on SATA 3Gb/s, and the host is SATA150. I just can't fathom how, with the processors and equipment that is involved, that there would be such a bottleneck. I just figured that if it is capable of 30-40MB/s for 5 out of 10 minutes, it should be able to sustain that, assuming the systems don't get bogged down by another system process or something...
In any case, I will check this out when I get to the house. Thanks for the info!

**vwgtiturbo** · 07-14-2006, 05:30 AM

Okay, I did the netcat testing from the link in the previous post, and here is what I have (after changing the sysctl settings yesterday, and not being able to change them back due to not knowing what the original values were):

540,688,384 bytes transferred = 4,325,507,072 bits

The times are as follows:
Real=12.943 sec
User=7.784 sec
Sys=4.072 sec

Using these numbers (with the 'Real' time above), I 'supposedly' get 334.1MB/s. Insane... Not exactly what I expect with Gigabit LAN, but I understand that Gigabit is a theoretical maximum, and with overheard and such, you will NEVER see that maximum, so...

Now, I am at a crossroads... I have to SATA powered systems with fairly beefy components, and am now looking for a bottleneck. I will ASSUME for now that it has to do with rsync and ssh, but... I wish there was a way to turn off SSH and see if that improved anything (damn the developer for making SSH the default!).

It could be rsync, but that would really be a bummer, as I just finished spending about a month writing a wicked backup script (well, I thought it was wicked, but it is really basic). I know, existing programs are out there, but I am a relative newby and I needed the learning experience, so...

**trickster** · 07-14-2006, 01:30 PM

If you tell us what version of Linnux you are using, and the files you modified, we might be able to help you put back the default settings.
Have you tried copying files between servers by using other protocols (ftp, smb, nfs, etc.)? That way you'd know if there really is a problem with the network, or if the problem is with the application you are currently using.

**vwgtiturbo** · 07-14-2006, 02:00 PM

I am using Suse 10.1, all patched up. I followed a guide to squeeze more performance out of your gigabit NIC here: http://www.enterprisenetworkingplane...le.php/3485486

But unfortunately, I didn't think to actually research what the commands did BEFORE I did them (now, at least I am not new enough to fall for the 'okay, now do "rm -rf /" to get your system's performance through the roof!'

)

Here is an excerpt:

-------------------------------------------------------------------
Configuring Linux and FreeBSD
For both Linux and FreeBSD we're using the sysctl utility. For all of the following options, entering the command 'sysctl variable=number' should do the trick. To view the current settings use: 'sysctl <variable name>'
Maximum window size:

Linux:
net.core.wmem_max=8388608

Default window size:

Linux, sending and receiving:
net.core.wmem_default = 65536
net.core.rmem_default = 65536

RFC 1323:
This enables the useful window scaling options defined in rfc1323, which allows the windows to dynamically get larger than we specified above.

Linux:
net.ipv4.tcp_window_scaling=1

Buffers:
When sending large amounts of data, we can run the operating system out of buffers. This option should be enabled before attempting to use the above settings. To increase the amount of "mbufs" available:

Linux:
net.ipv4.tcp_mem= 98304 131072 196608

These quick changes will skyrocket TCP performance. Afterwards we were able to run ttcp and attain around 895 Mb/s every time – quite an impressive data rate. There are other options available for adjusting the UDP datagram sizes as well, but we're mainly focusing on TCP here.

--------------------------------------------------------------

As far as other protocols, I have NOT done that one yet. I did some testing with netcat (as shown above) but I am not sure if my results are normal, or substandard, or...? I will try to set the server up for SMB this weekend, and see if I have any more luck (I only said SMB because I can get a really basic implementation of that done fairly quickly). As far as NFS, etc... I have zero experience with that, so it will take me a bit longer to get that done.

Thanks a lot, everyone, for your patience and assistance.