Iperf QUICK mode Ajay Tirumala, Les Cottrell

IEPM Home Page  IEPM Site Map  Network Monitoring  PingER  IEPM-BW Home IEPM Papers and Presentations

The problem

• For a TCP sender in an application, it is very difficult to guess when the connection is out of slowstart
• To counter this, bandwidth measurement applications (e.g.Iperf) will have to run for a considerable amount of time (steady state - congestion control/avoidance mode) to counter the effects of slowstart
• Though fairly accurate, these kind of tests take up considerable network bandwidth

Solution being explored

• Web100 provides access to kernel variables (substantial number) using which an application can GUESS when it's out of slowstart
• Even after slowstart, a connection may NOT be transferring data at a steady rate, due to transient network conditions OR due to protocol anomalies (e.g. Old TCP Reno)
• We can use the fact that the Congestion Window will be steady after slowstart
  • It increases exponentially during slowstart
  • If it's Reno, it will increase by one MSS every RTT (till it misses an ACK and drops it by 1/8th)
  • If it's Vegas, it will be fairly steady, oscillating around the threshold, unless new congestion is created
• The data transfer rate is fairly constant after slowstart
• We should be able to get a fairly accurate estimate of the bandwidth by measuring the throughput for a second or two after the connection is out of slowstart (and Congestion window seems steady)
• For this algorithm to work correctly, it is very important to set the appropriate (sufficient) window sizes at both the client and server. Instantaneous transfer rates for 1 GB of data from SLAC to ORNL with sufficient and less than sufficient window sizes is shown.
• Also, need to take care of multiple slow-starts. Check out the variation of Congestion Window and instantaneous bandwidth from SLAC to CERN
• Congestion Window along with throughput to various nodes from SLAC

Summary of observations and results

• Slowstart will be < 1 second for most networks and short RTTs < 50ms (even with high bandwidth)
• Slowstart can be upto 5 seconds when the delay is due to other streams congesting the bottleneck router (and hence increasing the delay)
  • This means that the problem of interfering tests remains
  • Though the probability of tests interfering comes down with the test duration being shortened
• Encouraging results : Throughput in QUICK mode matches throughput for running for 15-20 seconds closely
• Interesting result : A slow receiver blocking the sender usually results in lower throughput than in the slowstart phase. The reason behind this is the buffers initially get filled up quickly at the receiver, and if the receiver application is not able to match this rate, the receive side TCP advertises a low(sometimes 0) window size blocking the sender
• The tool was even tested with the server being on a dialup link. Need access to a Linux (with Web100) machine on a dialup to test client on a very slow link

Experiences with Web100

• Web100 has high utility from it's functional point of view.
• It consumes a lot of CPU, so it's advisable to run it in a separate thread
• It behaves well for a sampling rate of about 50 times a second
• Throughput of a 1.4 Gbps stream (transfer to the same machine) comes down to 485 Mbps if we use Web100 in the data transfer thread and sample one Web100 variable after sending each packet

Next Steps

• Integrate with BW tests
• Get results for longer tests/ parallel tests
• Get results for slow client/slow server

Explanation of differences/anomalies

• The node "node1.ece.rice.edu" is a Solaris machine. Solaris machines set the window size to minimal size, if the requested window size cannot be set. In our case the window size requested was 2M and the one set was 8K. By setting the window size successfully to 1M, we observed a bandwidth of 77.3 Mbps using Iperf for 20s and 80.7 Mbps for Iperf Quick mode
• Same as above with "node1.eecis.udel.edu". Obtained 41.1 Mbps - Iperf for 20s and 38.7 Mbps - Quick mode
• Same as above with "node1.utdallas.edu". Obtained 11.1 Mbps - Iperf for 20s and 9.0 Mbps - Quick mode


• Same as above with "node1.roma1.infn.it". Obtained 1.2 Mbps - Iperf for 20s and 1.0 Mbps - Quick mode
• Latency to this node is about 170 ms. Later tests indicate that it gets out of slowstart in ~3 sec. We assume packet loss during initial tests as the reason it takes so long to get out of slowstart. This assumption is also supported by the fact that it takes a long time to come out of the quick mode (1s after slow start) measurement. Iperf has blocking sends and TCP does not return the send call made at about 1s after the end of slow start.


• We assume a transient congestion while measuring to "node1.internet2.edu". Running the test later 4 times averaged 75.9 Mbps for Iperf for 20s and 78.5 Mbps for Quick mode
• Same with "node2.in2p3.fr". Running the test later 4 times averaged 26.9 Mbps for Iperf for 20s and 21.8 Mbps for Quick mode
• "node1.kek.jp" only allows a maximum window size of 128K. Though not a huge difference in the table above, setting the client window size also to 128K gave 3.7 Mbps by running Iperf for 20s and 3.8 Mbps for Quick mode
• The most interesting result is the one with ORNL. The bandwidth to ORNL is about 200 Mbps and the RTT is about 80ms. This makes the BW*delay product ~2 MB. Sometimes the Congestion Window will be double this value when it's just out of slow-start and it will return back to 2MB quickly. We must make provisions for this. Setting window sizes at both ends to 8MB resulted in 201 Mbps by running Iperf for 20s and 195 Mbps for Quick mode
• The same case with "node1.lanl.gov". Having a TCP window size of 12M on both sides resulted in 304 Mbps by running Iperf for 20s and a very close 318 Mbps for Quick mode