Report on individual activity to the PPDG for the quarter ending December, 2001
 
Les Cottrell, SLAC

Project: Network throughput performance

We put together a project for the SC2001 Bandwidth Challenge.
See http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2001/ for
more details. 
It included over 25 collaborating sites (including all the PPDG sites) to 
which we sent large amounts of bulk throughput from the SLAC/FNAL
booth at SC2001. We achieved over 1.6Gbps/sec throughput simultaneouly
to about 17 sites in 5 countries. We also demonstrated the 
effectiveness of QBSS
for very high speed links (2Gbits/sec). Following SC2001 we extended
and ruggedized the infrastructure put in place for the SC2001 bandwidth 
challenge and tentatively names the project IEPM-BW (Internet
End-to-end Performance measuremene - BandWidth). 
We now have about 30 sites in 8 countries, and are making 
regular measurements with ping, traceroute, bbcp (both memory to memory and 
disk to disk), bbftp and pipechar.

We are starting to analyze the data from these measurements. We made 
presentations on Achieving High throughput performance at the
ESCC meeting at ANL, and the Inaugural Internet 2 HENP networking
working group at Ann Arbor Michigan.
We made presentations on High throughput network performance measurements
to the ICFA / Standing Committee on Interregional Connectivity
(SCIC) at CERN, and at the Babar collaboration meeting at SLAC. We have also
made presentatins on QBSS, PingER futures, and Grid Monitoring at ESCC and the
Virtual Internet 2 Members meeting.

Early results from the IEPM-BW project indicate:

* Reasonable estimates of throughput can be obtained with 10 second 
iperf or bbcp measurements. This is typically much shorter than it 
takes to make a pipechar measurement. 
* In many cases it is not sufficient to simply increase the window 
size to achieve high throughput, multiple parallel streams are also critical. 
* Careful attention to window sizes and parallel streams in necessary. 
Improvements of between 5 and 60 times have been observed for the optimum 
window and stream settings compared to using a single stream and the 
default maximum window size.  
* It is also observed that there is an optimum window*number parallel 
streams beyond which performance does not increase, or may decrease, 
while cpu, packet loss increases. 
* Throughput can vary by an order of magnitude with time of day or 
day of week etc. 
* Roughly speaking one needs 1 MHz to provide 1 Mbit/sec on today's 
cpus and OSs. 
* The bbcp file copy rates from memory to memory are about 60+-20% 
of the iperf rates. 
* File copy rates disk to disk are typically about 90% of the memory 
to memory rates, for rates below 60Mbits/s, but can vary dramatically 
depending on disk performance, caching etc. Uncached disk 
performance typically tops out at between 4 and 8MBytes/sec. 
* In some cases (e.g. SLAC to CERN for BaBar Objectivity data) 
compression can improve throughput by over a factor of 2 on a reasonably 
high performance host (Sun 336MHz cpus). 
* When running high throughput applications, the RTT for other users 
can be noticeably increased, e.g. for SLAC to CERN the average 
increases from about 160 msec. to about 260 msec. 
* The impact of high throughput applications, on other applications 
requiring low latency, may be reduced by applying lower than best effort 
priority (Scavenger Service) to the high throughput applications' packets. 

We are in the process of improving the analysis and 
reporting/graphing/table tools. We are also
building tools to facilitate and automate the 
infrastructure management. This includes downloading of code, 
checking whether measurements are successful, gathering the remote 
configurations parameters (OS, cpu speed, code versions), 
understanding disk performance, verifying windows
and streams are set correctly. 

We plan create a web site organized to provide easy access to
all aspects of this project. We will also measure the 
impacts of compression, add and understand 
gridFTP and other bandwidth measurement tools, and compare and contrast
the various measurements.
Following this we will select a representative minimum subset of 
tools to make measurements with, improve the reporting/graphing/table 
tools and make the data available via the web. 
We also hope to tie together the measurements being made in the 
UK with the SLAC measurements so they appear more integrated  to the user.