Bulk Throughput Measurements - FNAL, Chicago

Bulk Throughput Measurements | Bulk Throughput Simulation | Windows vs. streams | Effect of load on RTT and loss | Bulk file transfer measurements


On October 30, 2001, measurements were made between from to Dcglnx was a 200MHz with a 100Mbps interface running Linux 2.4. The site was connected to Internet 2 via an OC3 (155Mbps) link. It was located at Fermilab near Chicago, Illinois. Pharlap was a Sun E4500 with 6*336MHz cpus and a GE interface running Solaris 5.8. SLAC had a 1Gbps link to the Stanford campus and from there a 622Mbps link to CalREN and Internet 2. The routes were symmetric using ESnet in both directions.. The ping response for 383 default length (64 Bytes) from SLAC to FNAL was min/avg/max (std) = 50.25/55.96/133.89 (15.94) ms. The pipechar from SLAC to dcglnx was also recorded. The window buffer sizes on pharlap are shown below:
ndd /dev/tcp tcp_max_buf = 4194304
;ndd /dev/tcp tcp_cwnd_max = 2097152
;ndd /dev/tcp tcp_xmit_hiwat = 16384
;ndd /dev/tcp tcp_recv_hiwat = 24576

The window buffer sizes on dcglnx are shown below:
more /proc/sys/net/core/wmem_max = 8388608
;more /proc/sys/net/core/rmem_max = 8388608
;more /proc/sys/net/core/rmem_default = 65536
;more /proc/sys/net/core/wmem_default = 65536
;more /proc/sys/net/ipv4/tcp_rmem = 4096 87380 4194304
;more /proc/sys/net/ipv4/tcp_wmem = 4096 65536 4194304

The performances shows that the maxima (the top 10% measurements) achieve over 17Mbits/s throughput. It is evident that to get the best performance one needs maultiple parallel streams. the apparent poor throughput maybe partially due to the low power (200MHz) of the dcglnx host. However, elsewhere we showed that it takes roughly 1 MHz/Mbps so 17Mbps would utilize only about 9% of the cpu. The relatively large value of the ping std, may indicate heavy congestion on the link. After working with the FNAL people it was discovered that there was a 100Mbps full-duplex half-duplex mismatch.  After fixing this the second graph below was obtained and the maxima were over 91Mbits/s an improvement of over a factor of 5.
SLAC-FNAL throughput

While setting up the NIC/switch configurations some measurements were made. These are shown in the MRTG plot below measured from the FNAL DMZ switch port.  The right hand peak is when the NIC/switch port were properly configured for full duplex. Part of the reason it does not show a higher value is since the measurements were made for 10 seconds on followed by 10 seconds off.  The plots at about 10:30 and 13:00 are for a 100Mbps half-duplex configuration. The difference is about a factor of 4.

Created October 31, 2001, last update November 1, 2001.
Comments to