IEPM

OSCARS Results

SLAC Home Page
Les Cottrell, SLAC and Chin Guok, ESnet

More measurements (Jan 2006)

Introduction

OSCARS the ESnet On-demand Secure Circuits and Advance Reservation System provides a prototype service that enables on-demand provisioning of guaranteed bandwidth secure circuits within ESnet. The reservations require a password and userid and are made via a Web site which provides access to a reservation web page. One can specify the time of a reservation as well as identify the source and destination, and a selection of port, DiffServ Code Point(DSCP) and protocol (e.g. TCP, UDP etc.). To access the page one has to be registered (currently the registration is by host) and has to have an OSCARS account. Each reservation is assigned an OSCARS circuit (OSCARS_nn) which we use below to identify the tests. One can read out the OSCARS policing counters for each reservation.

Measurements

OSCARS_76 Measurements

We accessed the OSCARS web page and set up a 10Mbits/s reservation from iepm-resp.slac.stanford.edu to port 5011 on pcgiga.cern.ch. Both hosts were set up with large maximum TCP windows (>20MBytes). We then set up iperf servers on pcgiga.cern.ch with 20MByte windows, one on port 5010 (best-effort), the other on port 5011 (OSCARS reservation for QoS).

Following this we ran an iperf/TCP client sending data for 60 seconds in a single stream from SLAC to CERN port 5010 (best-effort) on pcgiga.cern.ch. The RTT was about 170ms and we used a window of 10MBytes. We then repeated this test except to port 5011 (the OCSARS QoS port). The results indicate that with best effort (no limit on the bandwidth) we were able to achieve about 7.6Mbits/s in both cases. The OSCARS log showed no packets being policed.

We then repeated the test with 20 parallel streams and a 256KByte TCP send window. The results indicate that with best-effort we could achieve an aggregate of about 150Mbits/s, but with the OSCARS 10Mbits/s QoS reservation we were limited to 9.45Mbits/s, and individual streams typically were running at about 500Kbits/s. At the end of these multi-stream tests the OSCARS counters showed considerable number of policed packets.

We then logged onto OSCARS again and reduced the dedicated bandwith to 1 Mbits/s and repeated the tests with one stream. The results show there was no change in the best effort (port 5010) performance, however, as expected the performance on the OSCARS 1 Mbits/s QoS reservation was less than 1 Mbits/s (actual value 874Kbits/s).

Since iperf has difficulties providing 1 second interval updates we then repeated the 1Mbits/s reservation tests for 10 second durations with 1 second interval reporting with thrulay. Since thrulay only supports a single stream the measurements were made with a single stream only. The results indicate that with best effort service we achieved about 8.2Mbits/s. Thrulay also reports the Round Trip Time (RTT). Looking at the RTT it appears there was only elongation of the RTTs (indicating possible congestion) during the first second of the transfer. During this first second the throughput was also less than for the following 9 seconds. Presumably the first second effects are due to TCP's slow start. The test with the OSCARS 1Mbits/s reservation showed a similar behavior for the first second. In the second second the throughput measured was 5.95Mbits/s which is greater than the advertised limit of 1 Mbits/s. The remaining 8 seconds the troughputs ranged form 0.6 to 0.064Mbits/s. The aggregate throughput was 0.961Mbits/s which is, as expected, just under the advertised 1Mbits/s limit

OSCARS-91 Measurements

We repeated the thrulay measurements for a longer period (60 seconds) to see how well the throughput stabilized. For the 60 seconds measurement we captured plots of both the performance with, without QoS and with the QoS reservation kicking in. It can be seen that there is considerable instalbility for the first 20 seconds for the QoS path.

To see the longer term stability of throughput we repeated the QoS measurement for 600 seconds and analyzed the throughputs reported. The plot shows that the throughput is reasonably stable (median throughput =0.944Mbits/s, Inter Quartile Range (IQR)=0.26Mbits/s, min=0.332 Mbits/s, max=1.654Mbits/s). For the policing on Junipers, there is a buffer size setting that can be set to absorb instantaneous burst. Current we've set the default to 1MByte regardless of the policing rate. This potentially can skew slow duration transfers. If we set the buffer size extremely low, then the sending host must strictly police the flow to the requested bandwidth. We're still playing around to figure out what the buffer size should be in relation to the policing bandwidth. A suggestion that we got from Juniper was to set it at 1/10 of the policing rate. We'll need to investigate more, but I think the end goal is that it will be some fraction of the policing rate.

We also looked at thrulay's reported average RTT. As can be seen from the scatter plot there is a negative correlation.

At the completion of the above measurements we recorded the OSCARS policing counters.

We then set the the burst-size-limit down from 1MB to 100KB for the oscar_91 policer, as well as cleared out the counters. Following this we repeated the thrulay measurement for 8 minutes. The plot shows that the start up is much better behaved. The long term plot shows similar behavior to that with the burst-size-limit set to 1MB. the statistics for the two settings are compared in the table below:
burst-size-limitstartendmedianIQRminmax
1MByte102s600s0.94Mbits/s0.26Mbits/s0.33Mbits/s1.65Mbits/s
0.1MByte0s480s0.93Mbits/s0.26Mbits/s0.31Mbits/s1.389Mbits/s
We verified that when the OSCARS reservation had terminated we were able to get 8.193Mbits/s with thrulay to port 5011. It should also be noted that with more than 20 streams of iperf, the aggregate throughput increases and maxes out at about 450-500Mbits/s.

OSCARS - 96 Measurements, Jitter

We reserved 1Gbits/s of bandwidth between iepm-resp.slac.stanford.edu and iepmbw.bnl.gov using DifServ Code Point 1 to identify packets. We verified that the ping -Q option indeed set the DSCP/TOS bits by using tcpdump. Starting at 9:40pm we ran a ping with a reservation using DSCP=1 (TOS=4) (results):
9cottrell@iepm-resp:~>ping -Q 4 iepmbw.bnl.gov | tee /tmp/940pmrs
plus a ping with just best effort (results):
9cottrell@iepm-resp:~>ping -iepmbw.bnl.gov | tee /tmp/940pmbe
After running for 80 secnds we started an iperf with 40 streams for 600 seconds (results):
date ; iperf -c iepmbw.bnl.gov -p 5000 -t 600 -i 5 -w 256k -P 40 | tee /tmp/940p
We stopped the pings 60 seconds after the iperf finished. The results show that the reserved service gave lower jitter as measured by the RTT distributions' Inter Quartile Ratio, standard deviation, 5% and 95%, minimum and maximum. Based on measurements made during the daytime, we believe that in some cases the effect of iperf on the ping RTTs (without OSCARS QoS reservation) is much greater. In which case the improvement by applying the QoS is expected to be greater. the effect of the iperf streams on the ping RTTs
Revised June 29, 2005.
URL: http://www-iepm.slac.stanford.edu/dwmi/oscars/
Comments to iepm-l@slac.stanford.edu