IEPM

Bulk Thruput Measurements-Cal Tech

SLAC Home Page
Bulk throughput simulation | Windows vs. streams | Effect of load on RTT and loss | Bulk file transfer measurements

SLAC to Caltech

The first of the plots below shows the SLAC to Caltech performance as measured via ESnet in the fall of 2000. The second shows it as measured via Internet2/CalREN2. It is seen that one gets about a factor of 2 improvement when using Internet2. The path characterization measured on October 31, 2000 shows that pchar bottleneck bandwidth estimation badly underestimates the iperf thruput.
SLAC to Caltech bulk thruput SLAC to Caltech via Internet2
We also made measurements in November 2000 from a Pentium III running Linux with a Gbit interface at SLAC, via an OC12 (622Mbps) link provided by the experimental NTON network from SLAC to Caltech to another Pentium III host. Over this link we achieved about 500Mbits/s with a single stream and a window size of about 800KBytes or more. The results are shown to the right.

In August of 2001, we repeated the measurements over the production network. We used two different machines at Caltech, heppcn08.hep.caltech.edu (running Linux 2.2.18 with a 100Mbps Ethernet interface and a maximum receive buffer size of 64KBytes) and plato.cacr.caltech.edu (a 1GHz single cpu PC running Linux 2.4.5ac9 with a Syskonnect Gbps Ethernet interface and a maximum buffer/window size of 8MBytes). For heppcn08 we achieved maxima of about 85Mbits/s. The RTT was about 25msec. The pipechar indicated a bottleneck bandwith of about 100Mbps at the Caltech end. Due to the small window size available at heppcn08 multiple streams had to be used to achieve this throughput. For the measurement to plato, we only had a single iperf server port open on plato. This was enabled to have a maximum window size of 8MBytes. With such a window size on the server and using Linux 2.4, the actual window sizes used when requesting a 10 second transfer by the iperf client on pharlap for window sizes of 8k through 128k, as observed using snoop, are shown in the table below.
Window requested from iperf client (KBytes) Initial pharlap window in SYN (bytes) Initial plato window in SYN (Bytes) Final pharlap window (Bytes) Final plato window (Bytes)
887605840876066240
161752058401752064240
323358058403358064240
642*33500128*57962*34028128*49130
1284*32942128*57924*32942128*49130
The multipliers (e.g. 2* or 128*) in the above table are the effect of the window scaling factor advertized in the SYN packets at the start. It is seen that telling the client we require window sizes of 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1024k, 2048k and 4096k, with a Linux 2.4 iperf server, basically only uses 2 maximum receive window sizes of 64240 Byte (for 8k, 16k,and 32k requests) and 6MB otherwise. This is reflected in the throughputs measured, seen to the right, where 2 behaviors with streams are observed depending on the window sizes. It is seen that the maximum throughput of about 280Mbits/s is achieved with a single stream. The RTT was 24 msec. and the pipechar showed an almost identical path and again indicated the bottleneck was on the last hop at Caltech. The achievement of 280Mbps throughput for a path with an apparent pipechar estimated bottleneck bandwidth of about 100Mbps (or even 155Mbps if we exclude the last hop which we believe is Gbps), bears further investigation. Perhaps there are multiple paths with load balancing or pipechar is unreliable at these bandwidths. The fact that pipechar reports << for the first two links and the last link, that we know are 1 Gbps, makes us suspicious of the internmediate results. The cpu loads on the iperf client on pharlap (a 6 * 336 MHz Sun E4500) for the 10 second measurements were:

		user	sys	%(sys+user) MHz/Mbps	
average 	.18	5.9	58		.94
stdev		.1	3.5	3.2		.27
median	0.2	6.95	65		.97
max		.29	9.4	95		1.32
i.e. roughly 1 MHz/ Mbps.

Caltech to SLAC

We measured the iperf TCP throughput for various window sizes (between 16KB and 2MB) and numbers of streams from plato.cacr.caltech.edu to pharlap.slac.stanford.edu. The top 10 percentile measurements are in the range 297-348Mbits/s. It appears that one cannot get close to the maxima with a single stream and large windows alone. Multiple streams are needed also. The RTT is about 25msec. So if the bottleneck is 350Mbps then the predicted optimum window size is RTT*BW or about 1 Mbyte. The median MHz/Mbps was 235. We had concerns (see also the section on Measurement Duration in Bulk File Transfer Measurements) that the poor performance of large windows and single or a small number of streams might be an artifact of the measurement being insufficiently long to allow the single stream-large window flow to get up to spped. We therefore repeated the measurement for 50 second individual measurements. The result, seen to the right, indicates that for a 1MByte window the throughput for 2 or more streams reaches over 90% of the 50 second throughput in 10 seconds, and for one streams reaches about 70%. For a 2MByte window the throughput for 10 seconds reaches 90% of the throughput for 50 seconds for 6 or more streams. For a single stream the 10 second throughput si about half that for a 50 second measurement.

For the measurement from heppcn08.hep.caltech.edu to pharlap.slac.stanford.edu the maxima of the throughputs achieved was about 89Mbps. Thus we can use about 90% of the bandwidth of a fast Ethernet interface over the WAN. Further, this is almost a factor of 4 times worse than to plato.cacr.caltech.edu (which has a 1Gbit/s Ethernet) over the same route. This indicates the importance of the interface speeds of the hosts.

Novermber '01

It was observed that the performance from SLAC to Caltech If you plow through the rest of this section (it is basically a chronologic log of what I haver tried), you will see that I am suspicious that the performance issue is isolated to pharlap to Caltech and is between ESnet and Caltech (actually between ESnet at General Atomics in San Diego with UCSD). I doubt we will be able to get all the parties involved (ESnet etc.) together to fix that until we get the new OC12 fully working at SLAC, especially since it only affects pharlap at SLAC. Perhaps we can do something at the SLAC router (Gary?) but we are all buried under getting SC2001 running for the next 2 days. Unfortunately I do not have a well connected fast network test machine besides pharlap that we can put on the regular SLAC network.

The current route from SLAC to Caltech for all but one SLAC host is still via CalREN2, for example:

1cottrell@tersk07:~>traceroute plato.cacr.caltech.edu
traceroute: Warning: ckecksums disabled
traceroute to plato.cacr.caltech.edu (131.215.144.226), 30 hops max, 40 byte packets
 1  RTR-FARMCORE1A.SLAC.Stanford.EDU (134.79.127.7)  0.566 ms  0.412 ms  0.371 ms
 2  RTR-MSFC-DMZ.SLAC.Stanford.EDU (134.79.135.21)  0.479 ms  0.415 ms  0.385 ms
 3  192.68.191.83 (192.68.191.83)  0.403 ms  0.329 ms  0.274 ms
 4  STAN.POS.calren2.NET (171.64.1.213)  0.390 ms  0.401 ms  0.392 ms
 5  SUNV--STAN.POS.calren2.net (198.32.249.73)  0.744 ms  0.681 ms  0.645 ms
 6  C2-QANH-GSR-QSV-GSR.ATM.calren2.net (198.32.249.154)  21.032 ms  21.001 ms  20.979 ms
 7  UCI--QANH.POS.calren2.net (198.32.248.126)  21.655 ms  21.637 ms  21.766 ms
 8  UCR--UCI.POS.calren2.net (198.32.248.13)  22.999 ms  23.046 ms  23.136 ms
 9  CIT--UCR.POS.calren2.net (198.32.248.9)  24.719 ms  24.515 ms  24.590 ms
10  Caltech-CalREN.caltech.edu (192.41.208.50)  24.785 ms  24.544 ms  24.751 ms
11  CACR-rtr.ilan.caltech.edu (131.215.254.145)  25.359 ms  25.114 ms  25.012 ms
12  plato.cacr.caltech.edu (131.215.144.226)  24.833 ms  24.721 ms  24.644 ms
The route from Caltech is also over CalREN2 for both Pharlap (apart from the last hops) and the rest of SLAC:
traceroute to tersk08.slac.stanford.edu (134.79.125.198), 30 hops max, 38 byte packets
 1  defaultroute (131.215.144.254)  1.036 ms  0.472 ms  0.420 ms
 2  Booth-border.ilan.caltech.edu (131.215.254.254)  0.099 ms  0.090 ms  0.081 ms
 3  CalREN-Caltech.caltech.edu (192.41.208.49)  0.172 ms  0.161 ms  0.137 ms
 4  UCR--CIT.POS.calren2.net (198.32.248.10)  1.781 ms  1.745 ms  1.732 ms
 5  UCI--UCR.POS.calren2.net (198.32.248.14)  3.025 ms  2.974 ms  2.988 ms
 6  QANH--UCI.POS.calren2.net (198.32.248.125)  3.739 ms  3.669 ms  3.813 ms
 7  C2-QSV-GSR-QANH-GSR.ATM.calren2.net (198.32.249.153)  23.943 ms  24.075 ms  23.922 ms
 8  STAN--SUNV.POS.calren2.net (198.32.249.74)  24.409 ms  24.264 ms  24.439 ms
 9  i2-gateway.Stanford.EDU (171.64.1.214)  24.549 ms  24.471 ms  24.373 ms
10  * * *
11  * * *
12  TERSK08.SLAC.Stanford.EDU (134.79.125.198)  24.734 ms  24.704 ms *
Pharlap is a special case on the new (in test) ESnet OC12 link to SLAC for the duration of SC2001. After SC2001 we will work on bringing the rest of SLAC so it is able to use the OC12 to ESnet. We will probably maintain the routing to Caltech on CalREN2. Measurements from another well connected (GE to the LAN) host at SLAC (connected via CalREN2) show throughputs of between 150 and 200Mbits/s with large numbers of streams (40). Plato appears to have the large windows set properly:
more /proc/sys/net/core/wmem_max = 8388608
;more /proc/sys/net/core/rmem_max = 8388608
;more /proc/sys/net/core/rmem_default = 65536
;more /proc/sys/net/core/wmem_default = 65536
;more /proc/sys/net/ipv4/tcp_rmem = 4096        87380   4194304
;more /proc/sys/net/ipv4/tcp_wmem = 4096        65536   4194304
Pipechar from the rest of SLAC appears to indicate we should be able to achieve well over 100 Mbps:
2cottrell@tersk07:~>pipechar plato.cacr.caltech.edu
AFS Password: 
0: localhost [12 hops]
1: RTR-FARMCORE1A.SLAC.Stanford.EDU    (134.79.127.7)     0.13   0.49   1.15ms
2: RTR-MSFC-DMZ.SLAC.Stanford.EDU      (134.79.135.21)    0.45  20.65  21.58ms
3:                                     (192.68.191.83)    0.44   0.56   1.11ms
4: STAN.POS.calren2.NET                (171.64.1.213)     0.44   0.49   1.19ms
5: SUNV--STAN.POS.calren2.net          (198.32.249.73)    0.48   0.60   1.78ms
6: C2-QANH-GSR-QSV-GSR.ATM.calren2.net (198.32.249.154)   0.49   0.55  23.38ms
7: UCI--QANH.POS.calren2.net           (198.32.248.126)   0.45   0.48  23.44ms
8: UCR--UCI.POS.calren2.net            (198.32.248.13)    0.41   0.44  23.85ms
9: CIT--UCR.POS.calren2.net            (198.32.248.9)     0.51   0.56  28.79ms
10: Caltech-CalREN.caltech.edu          (192.41.208.50)   0.48   0.56  26.59ms
11: CACR-rtr.ilan.caltech.edu           (131.215.254.145)         0.53   0.50  27.26ms
12: plato.cacr.caltech.edu              (131.215.144.226)         0.42   0.40  21.19ms

PipeCharacter statistics: 94.11% reliable
From localhost:
|       558.140 Mbps GigE (1026.6667 Mbps)

1: RTR-FARMCORE1A.SLAC.Stanford.EDU(134.79.127.7)
|
|       163.996 Mbps unKnown link ??? congested bottleneck <73.9875% BW used>
2: RTR-MSFC-DMZ.SLAC.Stanford.EDU  (134.79.135.21)
|
|       166.690 Mbps unKnown link ??? congested bottleneck <73.8701% BW used>
3:                                 (192.68.191.83)
|
|       649.792 Mbps   OC12      <1.1286% BW used>
4: STAN.POS.calren2.NET            (171.64.1.213)
|
|       159.474 Mbps    OC3      <7.9832% BW used>
5: SUNV--STAN.POS.calren2.net      (198.32.249.73)
|
|       151.136 Mbps    OC3 ??? congested bottleneck <4.6167% BW used>
6: C2-QANH-GSR-QSV-GSR.ATM.calren2.net(198.32.249.154)
|
|       166.132 Mbps    OC3 ??? congested bottleneck <2.9969% BW used>
7: UCI--QANH.POS.calren2.net       (198.32.248.126)
|
|       626.979 Mbps   OC12      <9.3126% BW used>
8: UCR--UCI.POS.calren2.net        (198.32.248.13)
|
|       155.969 Mbps    OC3      <20.1170% BW used>
9: CIT--UCR.POS.calren2.net        (198.32.248.9)
|
|       159.510 Mbps    OC3      <6.8359% BW used>
10: Caltech-CalREN.caltech.edu      (192.41.208.50)
|
|       155.979 Mbps    OC3      <9.3156% BW used>
11: CACR-rtr.ilan.caltech.edu       (131.215.254.145)
|       142.518 Mbps OC3 (156.8175 Mbps)

12: plato.cacr.caltech.edu          (131.215.144.226)
On the other hand the pipechar from pharlap to Caltech seems to show a restriction between ESnet and UCSD (maybe a T3)
4cottrell@pharlap:~>pipechar plato.cacr.caltech.edu
AFS Password: 
/afs/slac.stanford.edu/package/netperf/bin/@sys/pipechar [Beta830-2K] : can't reach the host12[131.215.254.145] with max_ttl(13)
try to analyze partial path instead

0: localhost [12 hops]
1: RTR-MSFC-DMZ.SLAC.Stanford.EDU      (134.79.243.1)     0.31   0.63   1.18ms
2:                                     (192.68.191.146)   0.66   0.94   1.77ms
3: snv-pos-slac.es.net                 (134.55.209.1)     0.61  -0.33   2.05ms
4: gac-snv.es.net                      (134.55.208.178)   1.15   1.22  17.35ms
5: UCSD--esnet.ATM.calren2.net         (198.32.248.69)    1.89   2.19  25.78ms
6: USC--UCSD.POS.calren2.net           (198.32.248.33)    1.78   2.31  24.64ms
7: ISI--USC.POS.calren2.net            (198.32.248.26)    1.87   1.98  25.92ms
8: UCLA--ISI.POS.calren2.net           (198.32.248.30)    1.83   2.40  27.71ms
9: JPL--UCLA.POS.calren2.net           (198.32.248.2)     1.76   2.33  28.45ms
10: CIT--JPL.POS.calren2.net            (198.32.248.6)    1.95   2.43  31.20ms
11: Caltech-CalREN.caltech.edu          (192.41.208.50)   1.87   2.46  28.12ms
12: CACR-rtr.ilan.caltech.edu           (131.215.254.145)         1.88   2.45  33.85ms

PipeCharacter statistics: 94.11% reliable
From localhost:
|       233.766 Mbps GigE (1026.6667 Mbps)

1: RTR-MSFC-DMZ.SLAC.Stanford.EDU  (134.79.243.1)
|
|       157.295 Mbps    OC3      <6.8285% BW used>
2:                                 (192.68.191.146)
|
|       159.364 Mbps    OC3      <6.8285% BW used>
3: snv-pos-slac.es.net             (134.55.209.1)
|
|       156.687 Mbps    OC3      <39.0708% BW used>
4: gac-snv.es.net                  (134.55.208.178)
|
|       46.895 Mbps     T3       <6.0190% BW used>
5: UCSD--esnet.ATM.calren2.net     (198.32.248.69)
|
|       46.330 Mbps     T3       <6.0190% BW used>
6: USC--UCSD.POS.calren2.net       (198.32.248.33)
|
|       45.348 Mbps     T3       <4.7618% BW used>
7: ISI--USC.POS.calren2.net        (198.32.248.26)
|
|       46.041 Mbps     T3       <2.0331% BW used>
8: UCLA--ISI.POS.calren2.net       (198.32.248.30)
|
|       45.411 Mbps     T3       <3.9870% BW used>
9: JPL--UCLA.POS.calren2.net       (198.32.248.2)
|
|       46.911 Mbps     T3       <9.9847% BW used>
10: CIT--JPL.POS.calren2.net        (198.32.248.6)
|
|       45.415 Mbps     T3       <4.3011% BW used>
11: Caltech-CalREN.caltech.edu      (192.41.208.50)
|       38.318 Mbps T3 (45.7700 Mbps)

12: CACR-rtr.ilan.caltech.edu       (131.215.254.145)
So then the question is why do we get excellent performance (400Mbps) to multivac.sdsc.edu but poor performance to plato.cacr.caltech.edu. Looking at the route below, it appears that ESnet peers with SDSC at SNV (Sunnyvale where they peer with CalREN2) but with Caltech at GAC (Gerneral Atomics in San Diego).
5cottrell@pharlap:~>pipechar multivac.sdsc.edu
AFS Password: 
0: localhost [7 hops]
1: RTR-MSFC-DMZ.SLAC.Stanford.EDU      (134.79.243.1)     0.65  -2.78   1.45ms
2:                                     (192.68.191.146)   0.63   0.88   1.59ms
3: snv-pos-slac.es.net                 (134.55.209.1)     0.70  -1.65   2.02ms
4: snva-esnet.abilene.ucaid.edu        (198.32.11.93)     0.71   0.74   2.23ms
5: losa-snva.abilene.ucaid.edu         (198.32.8.18)      0.60   0.74   9.95ms
6: piranha.sdsc.edu                    (192.12.207.78)    0.62   0.63  28.73ms
7: multivac.sdsc.edu                   (132.249.20.57)    0.89   0.92  30.62ms

PipeCharacter statistics: 1.19% reliable
From localhost:
|       110.940 Mbps OC3 (157.6028 Mbps)

1: RTR-MSFC-DMZ.SLAC.Stanford.EDU  (134.79.243.1)
|
|       157.295 Mbps             <3.2357% BW used>
2:                                 (192.68.191.146)
|
|       159.364 Mbps             <10.6686% BW used>
3: snv-pos-slac.es.net             (134.55.209.1)
|
|       156.687 Mbps             <0.4249% BW used>
4: snva-esnet.abilene.ucaid.edu    (198.32.11.93)
|
|       159.474 Mbps             <15.0141% BW used>
5: losa-snva.abilene.ucaid.edu     (198.32.8.18 )
|
|       158.140 Mbps             <3.0693% BW used>
6: piranha.sdsc.edu                (192.12.207.78)
|       80.537 Mbps 100BT (101.2739 Mbps)

7: multivac.sdsc.edu               (132.249.20.57)
On November 12, 2001, we placed a static route into the router at SLAC that specifified any traffic coming from pharlap's network and destined for plato.cacr.caltech.edu should go to Stanford University. The throughput as a function of window size and streams was remeasured and the maxima were over 159Mbits/s.

SLAC to CALTECH REVISITED

Jerrod D. Williams-April 10, 2003 Tests to Caltech from SLAC were showing a very distinct oscillation and we are trying to develop ways of normalizing these trends. The first step in doing so is to accumulate data from different times of the day to identify/varify the nature of the variations we are seeing. I began this study by identifying the "interesting" parts of the day that show the variations. Noting that node1.cacr.caltech.edu is a machine on a University's network, I understood that traffic varies based on the school day and "normal" computer usage of the students. I identified the greater portion of a typical school day to be between 8am and 4pm on weekdays. Periodic tests during these hours will suffice our need for a typical days activity. I did not want to test the 8 and 4 o'clock hours because during this time, little work is being done, and students will be in a transitional state(entering/leaving campus etc.) so I chose times an hour after students may "arrive" on campus (9am) and an hour before students "leave" campus (3pm). The busiest time on campus, based on diurnal analyisis, analysis of the BW-tests results, and personal experience, was identified as the midday hour(12 noon). I also wanted to analyse some traffic at slower hours of the day and identified those times as 6am and 6pm.

tcpload.pl was used to run a series of iperf tests from SLAC to CALTECH at the above specified times on Monday April 7, 2003. Tests were all conducted roughly 30mins. past the hour and they took around 45mins to an hour to complete dependant upon Iperf server failures.

Results by the hours

6am Results
During the 6am hour, we were able to reach maximum throughputs of 404.9Mb/s. The optimum throughput I noticed would be 332Mb/s using a 512K window and 12 streams.

 

 

 

 

 

9am Results
During the 9am hour, we were able to reach maximum throughputs of 226.3Mb/s. Here, I'd select a 1024K window with 12 streams to deliver the optimum throughput of 178Mb/s.

 

 

 

 

 

 

12noon Results
During the 12noon hour, we were able to reach maximum throughputs of 161.57Mb/s. As expected, the Noon reading proved to be the worst of the day. The optimum settings at this time of day would be a 120K window with 20 streams. These settings will deliver throughputs of around 129.8Mb/s.

 

 

 

 

 

3pm Results
During the 3pm hour, we were able to reach maximum throughputs of 185.6Mb/s. A window of 512K and 40 streams gives you the optimum throughput of 140.9Mb/s.

 

 

 

 

 

 

6pm Results
During the 6pm hour, we were able to reach maximum throughputs of 295.5Mb/s. The optimum window size is 1024 with 20 streams during the 6 o'clock hour. These settings give you an optimum throghput of 235Mb/s.

 

 

 

 

 

 

 

 

Summary of Bulk Throughput Measurements


Created August 25, 2000, last update April 10, 2003.
Comments to iepm-l@slac.stanford.edu