IEPM

TCP Stacks Testbed


Hadrien Bullot and Les Cottrell. Created August 1st 2003, last update October 14th 2003

SLAC
Introduction :: Measurenent :: TCPLoad :: Kernels :: Networks :: Reverse-traffic :: SLAC-Chicago :: SLAC-UFL :: SLAC-Caltech :: SLAC-Cern :: SLAC-Manchester

Introduction
With the success of BaBar and the need to support multiple remote computer centers, the need for high performance between the remote computer centers and SLAC was imperative. To assist with understanding the performance, we set out to measure the bulk data flows to critical sites (BaBar regional centers etc.) using the production networks available today, to see how well they performed. We are now trying to tune new TCP stacks that may give even better performances.
We are testing the following stacks:

Measurements
In general, all the measurements was made from SLAC (dual-processor machines with a 2.4GHz or 3GHz, 1 GB of memories and with a Gigabit Ethernet card) to NSLabs at UFL, CERN, Caltech, UMich, CESnet. The bottleneck at SLAC is 622Mbps. In the plot all the IP were hidden for confidential purpose and the time is in PST. Currently each run is for about 20 minutes so we can see the performance when TCP is in the stable regime.
Currently at SLAC we have 3 machines (the name are hidden):
Also we added the following line in /etc/sysctl.conf for the sender and receiver. There are the default values, unless specified otherwise:
# increase Linux TCP buffer limits
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_mem = 33554432 33554432 33554432

# flush old route information
net.ipv4.route.flush = 1

For the Web100 kernel we added the following tuning:
# web100 tuning
net.ipv4.web100_default_wscale = 7
net.ipv4.web100_no_metrics_save = 1
net.ipv4.web100_sbufmode = 0 # Brian Tierney asked to turn off this option
net.ipv4.web100_rbufmode = 0 # Brian Tierney asked to turn off this option

# net100 tuning
net.ipv4.WAD_IFQ = 1
net.ipv4.WAD_FloydAIMD = 1

For Fast TCP, we have the following tuning:
#fast tuning
net.ipv4.tcp_vegas_cong_avoid = 1
net.ipv4.tcp_vegas_alpha = 200
net.ipv4.tcp_vegas_beta = 250
net.ipv4.tcp_vegas_gamma = 200
net.ipv4.tcp_vegas_fast_converge = 1
net.ipv4.tcp_vegas_pacing = 1

TCP Load
First we ran a TCP Load to know which window size is the best for TCP Reno. The tests was run from SLAC to CERN at different time of the day. The results show us that using a 1024k window size with 16 streams is a good compromise.

Kernels

Networks
Currently we have access to the following networks for the tests:

Reverse Traffic
We've made few tests to see if Fast TCP, based on Vegas TCP, has some problem to compete with a cross-traffic of Reno TCP (16 streams) coming from the receiver's side.

SLAC to Datatag in Chicago
We ran the tests from 11AM to 11PM PST. The bottleneck is at SLAC (622 Mbps). As explained before, we used 3 machines for these tests. The interval of Iperf is 5 sec. and the interval of RTT is 50 msec. We've made some UDP taffic to see how the stacks will behave and fill the avalaible bandwidth. We've made some change in Iperf in order to have sinusoIdal UDP.

We made some different tests for the cross-traffic.

SLAC to NSLabs in University of Florida
We ran the tests from 4PM to 4AM PST. The bottleneck is at UFL (467 Mbps). We used 2 machines for these tests. The interval of Iperf is 5 sec.

We made some different tests for the cross-traffic.

SLAC to Caltech
We ran the tests from 6PM to 6AM PST. Due to some upgrade on the link the bottleneck is unknow for the moment. We used 2 machines for these tests. The interval of Iperf is 5 sec.

We made some different tests for the cross-traffic.

SLAC to CERN
We ran the tests from 7AM to 10PM PST. We used 2 machines for these tests. The interval of Iperf is 5 sec.

SLAC to University of Manchester
We ran the tests from 7AM to 10PM PST. We used 2 machines for these tests. The interval of Iperf is 5 sec.
We made some different tests for the cross-traffic.