Bulk Throughput Measurements - University College London, England

Les Cottrell, created May 31 '02
Bulk Throughput Measurements | Bulk Throughput Simulation | Windows vs. streams | Effect of load on RTT and loss | Bulk file transfer measurements

Configurations

Peter Clarke of the HEP group at University College London set up a host (henceforth referred to as node1.ucl.ac.uk, this is not its real name which is with-held for security reasons). Node1 was an 447MHz PIII with a 100Mbits/s interface to a Cisco 65xx switch and then to the Manchester MAN and SuperJANET (2.5Gbps backbone, I think). It was running Linux 2.4.16 with a standard TCP stack. The windows/buffers were set as follows:
cat /proc/sys/net/core/wmem_max = 8388608
;cat /proc/sys/net/core/rmem_max = 8388608
;cat /proc/sys/net/core/rmem_default = 65535
;cat /proc/sys/net/core/wmem_default = 65535
;cat /proc/sys/net/ipv4/tcp_rmem = 4096 87380 174760
;cat /proc/sys/net/ipv4/tcp_wmem = 4096 16384 131072

At the SLAC end was a 1133 MHz PIII also running Linux 2.4. and with a 3COM GE interface to a Cisco 6509 and then via GE to a 622Mbps ESnet link. The TCP stack at the SLAC end was Web100. The windows/buffers settings were:
;cat /proc/sys/net/core/rmem_max = 8388608
;cat /proc/sys/net/core/rmem_default = 65536
;cat /proc/sys/net/core/wmem_default = 65536
;cat /proc/sys/net/ipv4/tcp_rmem = 4096 87380 4194304
;cat /proc/sys/net/ipv4/tcp_wmem = 4096 65536 4194304

Measurement methodology

We used tcpload.pl to make 10 second iperf TCP measurements. For each measurement we use a fixed window and number of parallel streams and measured client end (SLAC) cpu times, iperf throughput, ping responses (loaded) and also recorded various Web100 variables:
@vars=("StartTime",
               "PktsOut",        "DataBytesOut",
               "PktsRetrans",    "CongestionSignals",
               "SmoothedRTT",    "MinRTT",            "MaxRTT", "CurrentRTO",
               "SACKEnabled",    "NagleEnabled",
               "CurrentRwinSent","MaxRwinSent",       "MinRwinSent",
               "SndLimTimeRwin", "SndLimTimeCwnd",    "SndLimTimeSender");

Following each iperf measurement we ran ping for 10 seconds (unloaded) and recorded the responses. Following the above pair of a 10 second iperf measurement followed by 10 seconds of no iperf throughput, the stream size was changed and the pair repeated. When all selected window sizes had been measured, then a different number of streams was selected and the cycle repeated.

Results

A traceroute from SLAC to UCL is shown below:
3 192.68.191.146 (192.68.191.146) 0.514 ms (ttl=252!)
4 snv-pos-slac.es.net (134.55.209.1) [AS293 - Energy Sciences Network (ESnet)] 0.845 ms (ttl=251!)
5 chi-s-snv.es.net (134.55.205.102) [AS293 - Energy Sciences Network (ESnet)] 55.7 ms (ttl=250!)
6 nyc-s-chi.es.net (134.55.205.105) [AS293 - Energy Sciences Network (ESnet)] 68.8ms (ttl=249!)
7 193.62.157.213 (193.62.157.213) 68.8 ms (ttl=249)
8 146.97.37.89 (146.97.37.89) 149 ms (ttl=248)
9 po15-0.lond-scr.ja.net (146.97.35.137) 149 ms (ttl=247)
10 146.97.35.2 (146.97.35.2) 149 ms (ttl=246)
11 146.97.40.34 (146.97.40.34) 149 ms (ttl=245)
12 ucl.lmn.net.uk (194.83.101.6) 149 ms (ttl=244)
13 128.40.20.61 (128.40.20.61) 150 ms (ttl=243)
14 128.40.20.22 (128.40.20.22) 149 ms (ttl=242)
15 pc35.hep.ucl.ac.uk (128.40.4.35) 149 ms (ttl=241)

The throughput topped out at about 90Mbits/s (120 streams, 1024 MB window).

A plot of the throughputs vs streams and windows is seen below:

Comments to iepm-l@slac.stanford.edu