SLAC Home Page


SC2001 Bandwidth Challenge Proposal:
Bandwidth to the World
Formal Measurements
Sample Bulk Throughput Measurements
Cal Tech | CERN | Colorado | IN2P3 | INFN | Daresbury | Manchester | Rutherford | Stanford to Daresbury| ANL | BNL | FNAL | JLab | ORNL | LANL | LBL | NERSC | PNW | Rice | RIKEN | SDSC | SLAC LAN | SLAC To Stanford's Campus | SoX| TRIUMF | UFL | U of Michigan | UTDallas | Wisconsin

Status Reports
Oct 20, 2001, Oct 27, 2001, Nov 03, 2001

Historical Performance Reports
| Round trip time | Loss | Derived bandwidth | Conditional loss | IPDV | Duplicate packets

Performance Maps
Iperf | Ping RTT & Loss 

More on bulk throughput
Bulk throughput measurements | Bulk throughput simulation | Windows vs. streams | Effect of load on RTT and loss | Bulk file transfer measurements | QBSS measurements

Page Contents
Primary Contact | Site Contacts | Project Description | Detailed Technical Requirements

Primary contact

Dr. R. Les Cottrell, MS 97, Stanford Linear Accelerator Center (SLAC), 2575 Sand Hill Road, Menlo Park, California 94025, <>

Contact information for all collaborators

Emails were sent on September 21-23, 2001 to potential contacts for sites, inviting them to join the project. The contacts (those in red have agreed to join) ordered alphabetically by site are shown below:

Linda Winkler, ANL, US, <> + William E. Allcock []
Dantong Yu, BNL, Long Island, US, <>
Harvey Newman, Caltech, Pasadena, US, <> + Julian J. Bunn [] + Suresh Singh <>
Olivier Martin, CERN, Geneva, CH, <> + Sylvain Ravot []
Robin Tasker, Daresbury Lab, Liverpool, UK, <> + Kummer, PS (Paul) []
Jim Leighton, ESnet, Berkeley, US, <>
Ruth Pordes, FNAL, Chicago, US, <> + Frank Nagy <> + Phil DeMar <>
Andy Germain, GSFC, US, <> + George Uhl []
Jerome Bernier, IN2P3, Lyon, FR, <> + Dominique Boutigny []
Emanuele Leonardi, INFN, Rome, IT,  <>
Guy Almes, Internet 2, US, <> + Matt Zekauskas <> + Stanislav Shalunov <> + Ben Teitelbaum <>
Chip Watson, JLab, Newport News, US, <>
Yukio Karita, KEK, Tokyo, JP, <>, Teiji Nakamura <>
Wu-chun Feng, LANL, Los Alamos, US, <>, Mike Fisk <>
Bob Jacobsen, LBL, Berkeley, US, <>, Shane Canon <>
Tom Dunigan, ORNL, Oak Ridge, US, <> + Bill Wing <>
Richard Baraniuk, Rice University, <>, Rolf Riedi []
John Gordon, Rutherford Lab, Oxford, UK, <J.C.Gordon@RL.AC.UK> + Adye, TJ (Tim) [T.J.Adye@RL.AC.UK]
Reagan Moore, SDSC, San Diego, UA, <moore@SDSC.EDU> + Kevin Walsh [kwalsh@SDSC.EDU] + Arcot Rajasekar <sekar@SDSC.EDU>
Warren Matthews, SLAC, Menlo Park, US <> + Paola Grosso <> + Gary Buhrmaster <> + Connie Logg <> + Andy Hanushevsky <> + Jerrod Williams <> + Steffen Luitz <>
Warren Matthews, Stanford University, Palo Alto, US, Milt Mallory <>
William Smith, Sun Micro Systems [],  Rocky Snyder
Andrew Daviel, TRIUMF, Vancouver, CA, <>
Paul Avery, University of Florida, Gainesville, US,  <> + Gregory Goddard []
Thomas Hacker, University of Muchigan, <>
Joe Izen, University of Texas at Dallas, US, <>
Miron Livny, University of Wisconsin, Madison, US, <> + Paul Barford <> + Dave Plonka <>

Project description

The unprecedented avalanche of data already being generated by and for new and future High Energy and Nuclear Physics (HENP) experiments at Labs such as SLAC, FNAL, KEK and CERN is demanding new strategies for how the data is collected, shared, analyzed and presented. For example the SLAC BaBar experiment and JLab are each already collecting over a TByte/day, and BaBar expects to increase by a factor of 3 in the coming year. The Fermilab CDF and D0 experiments are ramping up to collect similar amounts of data, and the CERN LHC experiment expects to collect over ten million TBytes. The strategies being adopted to analyze and store this unprecedented amount of data is the coordinated deployment of Grid technologies such as those being developed for the Particle Physics Data Grid and the Grid Physics Network. It is anticipated that these technologies will be deployed at hundreds of institutes that will be able to search out and analyze information from an interconnected worldwide grid of tens of thousands of computers and storage devices. This in turn will require the ability to sustain over long periods the transfer of large amounts of data between collaborating sites with relatively low latency. The Bandwidth to the World project is designed to demonstrate the current data transfer capabilities to several sites with high performance links, worldwide. In a sense the site at SC2001 is acting like a HENP tier 0 or tier 1 site (an accelerator or major computation site) in distributing copies of the raw data to multiple replica sites. The demonstration will be over real live production networks with no efforts to manually limit other traffic. Since we believe we will be able to saturate our link to SciNet and control the router in our booth, which will be at one end of the congested link, we also hope to be able to investigate/demonstrate the effectiveness of QBone Scavenger Service (QBSS)  in managing competing traffic flows and on the response time of lower volume  interactive traffic on high performance links.

On the SC2001 floor we will have a few high performance Linux hosts with Gbit/sec network interfaces connected to a Cisco 65xx series Catalyst switch to be located in the SLAC/FNAL booth. The switch will have at least 2 * 1Gbps links to SCiNet and a built in router capability. The hosts will will run various high throughput applications including  iperf and bbcp publication, man pages) a secure peer-to-peer high performance copy program. These programs will be called from scripts to automate running multiple copies, gathering performance statistics, and reporting in real time and recording the results. We will also have scripts to gather SNMP data from the booth router. The over 20 remote sites are connected by various networks, including Internet 2, ESnet, JAnet, GARR, and Renater. We have identified and contacted sites and are identifying hosts that are suitable for the demonstration. The requirements for the hosts at the remote sites are fairly limited.

The information gathered will be recorded in files, and analyzed with various tools including Excel. Universal Time History (UTH) utility real time plots of the throughput and the aggregate throughput (see mock-up) to each remote site will be displayed on monitors in the SLAC/FNAL booth. On another monitor we will display  ping round trip times (RTT) to sites around the world from the Denver show floor, and on a third monitor we will show throughput from SLAC to various Grid and HENP sites wordlwide. Longer term analysis (non-real time) will be made to summarize and report on the demonstration and will be made publicly available via the web. The web URL for this project will be:

Detailed technical requirements

We will be using the Internet 2, ESnet, JAnet, GARR, Renater WANs and the CERN-STARTAP link. ESnet will have an OC48 from Denver to Sunnyvale. SLAC will have an OC12 to Sunnyvale. The SC2001 SLAC/FNAL booth will have 2*Gbps connections to SciNet.

Offsite resources will be at the sites listed in the table below. Each site will have one or more Unix hosts running an iperf server. We are measuring throughputs to many of the sites from SLAC. We also measure ping Round Trip Times (RTTs), losses, and derived throughputs among other metrics. Note that at the time the ESnet connection to SLAC was only OC3 (155Mbps). It will be upgraded to OC12 (622Mbps) before the start of  SC2001.

Logo Site Host OS MHz NIC Site connectivity Iperf BW from SLAC
ANL Linux 2.4.2 2*866 1000Mbps 622Mbps (ESnet) 118Mbps
Sep '01
go to home page BNL 
Solaris 5.8
Linux 2.2.18 i686
  155Mbps (ESnet) 55Mbps
Jul '01
Caltech Linux 2.4 1000 1000Mbps 622Mbps (I2) 320Mbps
Jul '01
CERN Solaris 5.6 2*360 100Mbps 155Mbps 80Mbps
Jul '01
Council for the Central Laboratory of the Research Councils Daresbury Linux 2.4.2-2 996.315     38Mbps
Sep '01
FNAL Linux 2.4 200 100Mbps 155Mbps (ESnet) 91Mbps 1 Nov '01
NASA Meatball GSFC FreeBSD 3.5 933 100Mbps (NASA) 20Mbps Oct '01, probs w' ssh
IN2P3 Solaris 5.7 4*400 1000Mbps 155Mbps 105Mbps
Jul '01
INFN Solaris 5.6 4*400 100Mbps (GARR) 30Mbps
internet2_logo Internet 2,
FreeBSD 4.3 1000 1000Mbps 2.5Gbps (I2) 230Mbps / 30Mbps Oct'01
Jefferson Lab JLab Linux 2.4 1495 100Mbps 155Mbps (ESnet) 58Mbps Oct-01
KEK Linux 2.2 451 10Mbps 10Mbps 8.8Mbps / 3.6Mbps Oct-01
Los Alamos National Laboratory LANL Linux 2.4 933 1000Mbps 622Mbps (ESnet) 400Mbps Nov 5 '01
Berkeley Lab LBL Linux 2.2 700   622Mbps (ESnet) 60Mbps
NERSC Solaris 5.6 6*336 1000Mbps 622 Mbps (ESnet) 96Mbps  / 60 Mbps (Oct '01)
Linux 2.2.18

AIX 4.3
400 1000Mbps 622Mbps (ESnet) 90Mbps
Sep '01
Council for the Central Laboratory of the Research Councils RAL Linux 2.2.16-3 604 1000Mbps 622Mbps (ESnet/JAnet) 38Mbps
Sep '01
Rice University Solaris 5.7 2*400 100Mbps   Inbound ports blocked (10/19/01) / 30Mbps (Oct-19 '01)
SDSC - a unit of UC San Diego SDSC (E6000), (Ultra 4)
Solaris 5.7.
Solaris 5.8
1000Mbps 622Mbps (Abilene),
622 Mbps (vBNS),
155Mbps (ESnet)
118Mbps  Sep-01 / 457Mbps, Oct-01
SLAC Solaris 5.8 6*336 1000Mbps 155Mbps (ESnet),
622Mbps (I2)
Linux 2.2 300 100Mbps 622Mbps (I2) 80Mbps
Sun Micro Systems   Solaris 5.8        
TRIUMF LOGO TRIUMF Linux 2.2.19 871 100Mbps 22Mbps (BCnet/CAnet 3) 16Mbps
U Florida,
Linux 2.4 800 1000Mbps 155Mbps (I2) 90Mbps
Sep '01
Back to the UTD homepage UT Dallas Solaris 5.7 2*400 100Mbps (I2) 40Mbps
Sep '01
U Michigan Linux 2.4 2*797 100Mbps (I2) 84Mbps Nov 05 '01
University of

Wisconsin-Madison U Wisconsin Linux 2.4 1000 1000Mbps OC12 (I2) 90Mbps Oct '01

We will be located in the SLAC/FNAL booth.

We only require IP based communications.

No specialized on-show floor equipment is needed for the demonstration.

Created September 21, 2001, last update November 9, 2001.
Comments to