Bulk Thruput Measurements-ANL

Bulk Thruput Measurements | Bulk throughput simulation | Windows vs. streams | Effect of load on RTT and loss | Bulk file transfer measurements

SLAC to ANL, Chicago

The plot below shows the thruput from SLAC to ANL on September 7, 2000, when the SLAC link to ESnet was 43Mbps T3 ATM link with an effective capacity (after ATM tax, IPv6 and VoIP testbed PVCs were subtracted of about 30Mbps). The second plot shows the thruput on October 27, 2000 after the link was upgraded to OC3. The thruput is seen to increase by over a factor of 2. The path characterization shows that the bottleneck bandwidth of 53Mbits/s is similar to the iperf meaured thruput.
SLAC to ANL bulk thruput

Iperf throughput from SLAC to ANL, Sep 28 2001 Further measurements were made on September 26, 2001 from pharlap.slac.stanford.edu to wiggum.mcs.anl.gov. Pharlap was a Sun E4500 with 6*226MHz cpus anda GE interface running Solaris 5.8. Wiggum was a 2*866 Mhz Linux 2.4.2 host with a GE interface. At this time the route was via ESnet, SLAC had an OC3 link and ANL an OC12. Pipechar indicates possible congestion at the ANL border. The throughput results are shown to the right. The maxima (i.e. top 10% of the throughput measurements) were over 118 Mbits/s and were achieved for 8 or more streams for window sizes of 64 KBytes or more. This behavior (i.e. improvements as increase the window from 8KBytes to 64 kBytes, followed by similar throughputs for window sizes of 64KBytes or more is typical of Linux 2.4 TCP stacks, the borderline being due to the window scaling factor).

Pharlap's link to ESnet was upgraded to an OC12 (622Mbps) Packet over Sonet (PoS) connection from the OC3 ATM connection on November 5 '01. We therefore repeated the measurements on November 6 '01, with the result below shown to the right. The reason for the curves for window size requests of 64MBytes or more being very similar is since the maximum receiver size window at ANL was 64KBytes. Pharlap's windows were set to:
ndd /dev/tcp tcp_max_buf = 4194304
;ndd /dev/tcp tcp_cwnd_max = 2097152
;ndd /dev/tcp tcp_xmit_hiwat = 16384
;ndd /dev/tcp tcp_recv_hiwat = 24576

On November 7th, Bill Allcock of ANL increased the windows at ANL to:
more /proc/sys/net/core/wmem_max = 8388608
;more /proc/sys/net/core/rmem_max = 8388608
;more /proc/sys/net/core/rmem_default = 65536
;more /proc/sys/net/core/wmem_default = 65536
;more /proc/sys/net/ipv4/tcp_rmem = 4096 87380 4194304
;more /proc/sys/net/ipv4/tcp_wmem = 4096 65536 4194304

Therefore on November 8th we repeated the measurements once again with the new window size at ANL to see what the effect was. A very similar graph was obtained as shown below. The strong resemblance between these two graphs for window sizes of 64KB and above, cast suspicion on whether the windows were being set correctly. We therefore used the Solaris snoop function to trace packets. Looking at the first 2 packets (the SYN from pharlap and the SYN/ACK from wiggum) seen below, it is evident that the window scale factor (wscale) from wiggum is 0 which means the maximum window size advertosed by the receiver is 64KBytes.
The iperf command issued on pharlap was:

8cottrell@pharlap:~>iperf -c wiggum.mcs.anl.gov -w 512k -t 1 -p 5008
------------------------------------------------------------
Client connecting to wiggum.mcs.anl.gov, TCP port 5008
TCP window size: 512 KByte
------------------------------------------------------------
[ 4] local 134.79.240.26 port 35544 connected with 140.221.11.99 port 5008
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.1 sec 1.6 MBytes 11.8 Mbits/sec

Using ps -efl on wiggum we identified that the iperf server on port 5008 had a window size of 512kBytes:

000 S cottrell 32479 1 0 69 0 - 2017 rt_sig Nov06 ? 00:00:00 /nfs/dsl-homes02/cottrell/package/iperf-1.2/iperf -s -w 512k -p 5008

The snoop output observed on pharlap is shown below, and indicates the wscale factor advertized in its opening (SNY/ACK) packet is 1. This would only allow a maximum window of 128KBytes:

14cottrell@pharlap:~>sudo snoop -i /tmp/anldump2 | head

1 0.00000 PHARLAP.SLAC.Stanford.EDU -> wiggum.mcs.anl.gov TCP D=5008 S=35544 Syn Seq=3858732054 Len=0 Win=32850 Options=<nop,wscale 4,nop,nop,tstamp 833502290 0,nop,nop,sackOK,mss 1460>

2 0.05013 wiggum.mcs.anl.gov -> PHARLAP.SLAC.Stanford.EDU TCP D=35544 S=5008 Syn Ack=3858732055 Seq=3416717703 Len=0 Win=5840 Options=<mss 1460,nop,nop,sackOK,nop,wscale 1>

Bigger windows

To see whether we could coud achieve optimal throughputs with only one stream we repeated the measurement on August 2 '02 with windows up to the mximum configured (8MBytes). For these measurements Nagle was disabled and SACK enabled at both ends. We also used the Web100 instrumented TCP stack to measure various TCP parameters such as smoothed RTT (SRTT) and Congestion signals (a congestion event is one or more packet losses within a single round trip time). In order to reduce the utilization of host resources we limited the product of windows*streams to 35MBytes and made no measurements for when this product was exceeded. The results below (we use a log x scale to better illustrate the behavior of small number of streams) show that even with a window size of 8MBytes, one can more than double the throughput by using multiple streams. We also plotted the smoothed RTT and congestion signals vs. the throughput. The lines are moving averages of the last 10 points (in Mbits/s) to help guide the eye. It is seen that as the throughput starts to exceed 300 Mbits/sec the smoothed RTT increases by up to a factor of 5.

Created August 25, 2000, last update August 2, 2002.
Comments to iepm-l@slac.stanford.edu