TCP Stacks on production links Fabrizio Coccetti and Les Cottrell. Created 14 April '03, last update 14 April '03

IEPM Home Page  IEPM Site Map  Network Monitoring  PingER  IEPM-BW Home IEPM Papers and Presentations

Introduction

With the success of BaBar and the need to support multiple remote computer centers, the need for high performance between the remote computer centers and SLAC was imperative. To assist with understanding the performance, we set out to measure the bulk data flows to critical sites (BaBar regional centers etc.) using the production networks available today, to see how well they performed.  We are now trying to tune new TCP stacks that may give even better performances.
We tested the following stacks:

• HighSpeed TCP. It is part of the web100 package. It is a modification to TCP's congestion control mechanism for use with TCP connections with large congestion windows, more details at the HighSpeed TCP home page.
• Fast TCP.  This protocol is developed at the California Institute of Technology; read more at the Fast TCP home page.
• Scalable TCP. It is a simple change to the traditional TCP congestion control algorithm: the increase after dropping is exponential instead of linear. More information can be found at the Scalable TCP principal web page.

Single streams measurements

• Summary of single stream tests, for High Speed TCP, Fast TCP and Scalable TCP
• Summary: on high speed/long delay links, for a single stream HS, Scalable and FAST almost always (the one excpetion was Reno outperformed FAST on a link with a large variation in RTT) outperform Reno in the steady state (i.e. after they have been running for some time, say over 10 times the initial slow start time).
• High Speed TCP single stream versus one Reno TCP single stream [excel spreadsheet]
• Scalable TCP single stream versus one Reno TCP single stream [excel spreadsheet]
• a Fast TCP single stream versus a Reno TCP single stream   [excel spreadsheet]
  • Summary: On multiple paths (SLAC to CERN, SLAC to NIKHEF, SLAC to Caltech, SLAC to APAN in Japan), FAST appears to outperform Reno, except in the case of Caltech where the large variation in RTTs caused Reno to outperform FAST.

Multiple streams measurements

• a Scalable TCP single stream versus Reno TCP multiple streams [excel spreadsheet]
• Summary: For a single stream between SLAC & CERN Scalable outperforms Reno by up to a factor of 3. For 5 Reno streams or more, Reno outperforms Scalable. These measurements were only made for 60 seconds and by looking at the single stream Reno results in Fast TCP on production links, a Fast TCP single stream versus Reno TCP multiple streams it appears these measurements may not be representative of the steady state. The RTT to CERN was about 228ms and the bottleneck capacity was about 600Mbits/s.
• a Fast TCP single stream versus Reno TCP multiple streams [excel spreadsheet]
• Summary: It appears that the FAST stream (on a particular path - SLAC to CERN with 200ms RTT with 8MByte windows - with a 622Mbits/s capacity bottleneck, no RED) gets about 215 +- 15 Mbits/s fairly consistently (FAST against 1 Reno stream got 244+-80Mbits/s) while as one adds more Reno streams, Reno increased from 87Mbits/s (1 stream Reno) to about 150Mbits/s (8 streams of Reno, in this case FAST got 200+- 100Mbits/s, so FAST was possibly backing off). This also identifies that the achievable bandwidth (8 streams Reno + 1 stream FAST) was at least 350Mbits/s if one takes the averages, or about 450Mbits/s if one takes the peaks. It is also noticeable that congestion events for the various streams often appear to synchronize.

Various measurements

• Fast and Reno TCP varying the window size between SLAC and CERN [excel spreadsheet]
• Summary: The optimum maximum window size from SLAC to CERN was 16MBytes. For a 16MByte maximum window size, a single Reno stream achieved 220+-60Mbits/s while FAST achieved ~295+-64Mbits/s. However when the FAST competed against Reno, then FAST achieved a similar thoughput (285+-77Mbits/s) while Reno achieved only 45+-30Mbits/s or roughly 5 times less than when running on its own. This may be indicative of lack of "Max-min" fairness since FAST appears to increase its share of the bandwidth by decreasing that of Reno. Table summarizing results.
• Installation Notes