TCP Stacks Testbed

With the success of BaBar and the need to support multiple remote computer centers, the need for high performance between the remote computer centers and SLAC was imperative. To assist with understanding the performance, we set out to measure the bulk data flows to critical sites (BaBar regional centers etc.) using the production networks available today, to see how well they performed. We are now trying to tune new TCP stacks that may give even better performances.
We are testing the following stacks:

• S-TCP: It is a simple change to the traditional TCP congestion control algorithm: the increase after dropping is exponential instead of linear. This protocol is developed at the Communication Engineering Lab of Cambridge University.
  Contact: Tom Kelly {ctk21 AT cam.ac.uk}
• Fast TCP: This protocol is based on Vegas TCP. It uses queuing delay, rather than loss probability, as the main measure of congestion. It is developed at the California Institute of Technology.
  Contact: Cheng Jin {chengjin AT cs.caltech.edu}
• HS-TCP: It is part of the web100 package. It is a modification to TCP's congestion control mechanism for use with TCP connections with large congestion windows. They use a table to indicate how much to increase the congestion window by when an ACK is received.
  Contact: Tom Dunigan {thd AT ornl.gov}
• HSTCP-LP: They develop mechanisms that provide "better-than-best-effort" service. The aim is to utilize only the excess network bandwidth as compared to the "fair share" of bandwidth as targeted by TCP. It is developed by Rice University.
  Contact: Aleksandar Kuzmanovic {akuzma AT rice.edu}
• H-TCP: The protocol makes improvement in responsiveness possible - while maintaining steady-state throughput - via alternative strategies. This protocol is developed by a team at Hamilton Institute.
  Contact: Douglas Leith {doug.leith AT may.ie}
• Bic-TCP (formerly BI-TCP): The protocol combines two schemes called additive increase and binary search increase. When the congestion window is large, additive increase with a large increment ensures linear RTT fairness as well as good scalability. Under small congestion windows, binary search increase is designed to provide TCP friendliness. This protocol is developed by a team at NCSU.
  Contact: Injong Rhee {rhee AT eos.ncsu.edu}
• Westwood+ TCP: They continuously estimate, at the TCP sender, the packet rate of the connection by monitoring the ACK reception rate. The estimated connection rate is then used to compute congestion window and slow start threshold settings after a timeout or a three duplicate ACKs. This protocol is developed at Politecnico di Bari.
  Contact: Saverio Mascolo {mascolo AT poliba.it}

In general, all the measurements was made from SLAC (dual-processor machines with a 2.4GHz or 3GHz, 1 GB of memories and with a Gigabit Ethernet card) to NSLabs at UFL, CERN, Caltech, UMich, CESnet. The bottleneck at SLAC is 622Mbps. In the plot all the IP were hidden for confidential purpose and the time is in PST. Currently each run is for about 20 minutes so we can see the performance when TCP is in the stable regime.
Currently at SLAC we have 3 machines (the name are hidden):

• Machine 1: We run Reno TCP or UDP traffic from this machine.
• Machine 2: We run all the "advanced TCP stack" from this machine.
• Machine 3: We run ping from this machine.

Also we added the following line in /etc/sysctl.conf for the sender and receiver. There are the default values, unless specified otherwise:

# increase Linux TCP buffer limits
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_mem = 33554432 33554432 33554432

# flush old route information
net.ipv4.route.flush = 1

For the Web100 kernel we added the following tuning:

# web100 tuning
net.ipv4.web100_default_wscale = 7
net.ipv4.web100_no_metrics_save = 1
net.ipv4.web100_sbufmode = 0 # Brian Tierney asked to turn off this option
net.ipv4.web100_rbufmode = 0 # Brian Tierney asked to turn off this option

# net100 tuning
net.ipv4.WAD_IFQ = 1
net.ipv4.WAD_FloydAIMD = 1

For Fast TCP, we have the following tuning:

#fast tuning
net.ipv4.tcp_vegas_cong_avoid = 1
net.ipv4.tcp_vegas_alpha = 200
net.ipv4.tcp_vegas_beta = 250
net.ipv4.tcp_vegas_gamma = 200
net.ipv4.tcp_vegas_fast_converge = 1
net.ipv4.tcp_vegas_pacing = 1

First we ran a TCP Load to know which window size is the best for TCP Reno. The tests was run from SLAC to CERN at different time of the day. The results show us that using a 1024k window size with 16 streams is a good compromise.

• parallel-TCP: we used a recent Linux kernel: kernel-2.4.22. As we decided before with the TCPLoad, we run TCP Reno with 16 streams
• S-TCP: We are using a GNU/Linux kernel 2.4.19 with the scalable 0.5.1 patch.
• Fast TCP: We are using a GNU/Linux kernel 2.4.21 with a beta version of the stack (released August 1st 2003).
• HS-TCP: We are using a GNU/Linux kernel 2.4.22 whith the Alpha 2.3 web100 patch.
• HSTCP-LP: The kernel is a GNU/Linux 2.4.22 with web100. It uses the FloyAIMD from Net100 and the implementation for TCP-LP.
• H-TCP: We are using a 2.4.22 kernel with the web100 patch and the HTCP patch.
• Bic-TCP: Bic-TCP uses a patched 2.4.22 GNU/Linux kernel.
• Westwood+ TCP: A developper is installing the kernel at SLAC.

Currently we have access to the following networks for the tests:

• UFL: There is a bottleneck of 467 Mbps.
• Nikhef: They have only one machine.
• CSNet: The throughput is around 230 Mbps. There is only one machine.
• INFN: The throughput is around 25 Mbps. There is 2 machines.
• Datatag: The bottleneck for Datatag at Chicago is at SLAC (622 Mbps). There is 5 machines in Chicago and 5 in Geneva.
• CERN: There is one machine and the throughput is around 550Mbps.
• ORNL: We have 2 machines
• LBL: We have 2 machines and the bottleneck is at SLAC.
• UMich: We have 2 machines. In testing From 11:30PM to 5AM. & 4PM to 10PM.
• Caltech: We have two machines.
• Manchester: We have two machines.

We've made few tests to see if Fast TCP, based on Vegas TCP, has some problem to compete with a cross-traffic of Reno TCP (16 streams) coming from the receiver's side.

We ran the tests from 11AM to 11PM PST. The bottleneck is at SLAC (622 Mbps). As explained before, we used 3 machines for these tests. The interval of Iperf is 5 sec. and the interval of RTT is 50 msec. We've made some UDP taffic to see how the stacks will behave and fill the avalaible bandwidth. We've made some change in Iperf in order to have sinusoIdal UDP.

We made some different tests for the cross-traffic.

We ran the tests from 4PM to 4AM PST. The bottleneck is at UFL (467 Mbps). We used 2 machines for these tests. The interval of Iperf is 5 sec.

We made some different tests for the cross-traffic.

We ran the tests from 6PM to 6AM PST. Due to some upgrade on the link the bottleneck is unknow for the moment. We used 2 machines for these tests. The interval of Iperf is 5 sec.

We made some different tests for the cross-traffic.

We ran the tests from 7AM to 10PM PST. We used 2 machines for these tests. The interval of Iperf is 5 sec.

We ran the tests from 7AM to 10PM PST. We used 2 machines for these tests. The interval of Iperf is 5 sec.
We made some different tests for the cross-traffic.