Bulk Throughput Measurements - SoX

Bulk Throughput Measurements | Bulk Throughput Simulation | Windows vs. streams | Effect of load on RTT and loss | Bulk file transfer measurements

SLAC to SoX and SoX to SLAC

On October 3, 2001, measurements were made between pharlap.slac.stanford.edu and gigatcp1.internet1.edu (199.77.194.26) located at SoX (Internet 2 Southern eXchange GigaPoP) in Atlanta. Phalap was a Sun E4500 with 6*336MHz cpus and a GE interface running Solaris 5.8. SLAC had a 1Gbps link to the Stanford campus and from there a 622Mbps link to CalREN and Internet 2. Gigatcp1 was a 1000MHz PC cpu running FreeBSD 4.3 with a GE interface and connected to Internet 2 at 2.5Gbps. The route was set up in both directions to use Internet 2. The ping response from SLAC to SoX was min/avg/max = 55/58/1830 median=55 msec. The pipechar from SLAC to SoX was also recorded. The window buffer sizes on pharlap are shown below:

ndd /dev/tcp tcp_max_buf = 4194304
;ndd /dev/tcp tcp_cwnd_max = 2097152
;ndd /dev/tcp tcp_xmit_hiwat = 16384
;ndd /dev/tcp tcp_recv_hiwat = 24576

The window buffer sizes on gigatcp1 were as shown below:

sysctl kern.ipc.maxsockbuf = kern.ipc.maxsockbuf: 20480000
;sysctl net.inet.tcp.rfc1323 = net.inet.tcp.rfc1323: 1
;sysctl net.inet.tcp.sendspace = net.inet.tcp.sendspace: 1024000
;sysctl net.inet.tcp.recvspace = net.inet.tcp.recvspace: 1024000

Unfortunately I was unable to install iperf on gigatcp1 with multi thread support (the host appears to lack pthreads support), so we were unable to make tests with multiple streams.

The performances (seen below) are very asymmetric, throughput being a factor of 8 better from SLAC to SoX than from SoX to SLAC. This is not understood. I looked with tcpdump on pharlap at the iperf packets coming from 199.77.194.26 the wscale is set to 7, and the window 65535 in the SYN packet. The SYN ACK from pharlap has a wscale of 7 and a window of 32772. The MSS advertised by 199.77.194.26 is 4430 bytes and pharlap advertises 1460 bytes. The don't fragment bit is set on most packets in both directions.

Created October 6, 2001, last update October 6, 2001.
Comments to iepm-l@slac.stanford.edu