Project Title: INCITE: Edge-based Traffic Processing and Inference
for High-Performance Networks
2. SLAC - Current Accomplishments (through April 2003)
ABwE – Available Bandwidth Estimator (Tasks 2, 4)
We have installed, configured and
tested a new method for
estimation (ABwE). During practical measurements, which we started
in November 2002, we have collected and evaluated hundreds of monitoring
results and studied
the behavior of this tool in real network conditions. In the past few months we
have started to compare these results to the results of other tools used for
network monitoring of bandwidth, particularly to the Iperf type of
Iperf measurement is very popular and it looks relatively simple. It is widely
used by many ISP's or network specialists for different types of network
Unfortunately, the Iperf results can be misleading
because they are very dependent on the parameters. Setting the optimal parameters is quite a delicate process.
Such “tuning” often uses quite sophisticated methods. Some of these methods
have been developed at SLAC and some of the methods are built into other
network measurement tools, for example the net test-2 tool
developed at NERSC. During
our analysis we discovered that even if the parameters
have been setup by these methods or by an expert, after some
time (several hours or days) they could become
invalid. This is because the load on the path plays an important role and the
path itself can change. On occasions the path between two hosts can change
quite often. Iperf is a TCP type tool and we know that all TCP applications are
dependent not only on the bandwidth between source and destination but also on
the load in the reverse path. For example, setting a large window size to
achieve good performance might be counter-productive on heavily loaded lines. The
Iperf results between well and poorly set parameters could differ by several
Instead of using of our own Iperf measurements, we compared
the ABwE results with the iperf results from the IEPM-BW project. The source
(Iperf client) send
s as many
packets as it can. The
aggregated traffic via TCP
applications. If the line is
relatively empty, the aggregated traffic is approaching
the real capacity of the path and the estimate is approximately correct.
However, if the line is
busy, the Iperf
packets share this path with
other user traffic ( cross-traffic)
and the Iperf result will probably an over-estimate of the available
This is because the capability to send packets to the line
(the "strength") of all applications is not the same.
Iperf is set to use
"the maximum of the possible" so it will
grab all available bandwidth on the path. And in some cases, because
Iperf is very aggressive, it can even suppress some light type applications
such as interactive traffic or web traffic and via this suppression to obtain a
little bit more bandwidth than only the free part of the path capacity. This principle
(“grabbing all bandwidth and even a little bit more”) has been verified by
several people in the past on experimental paths with one or two hops path.
However, how much Iperf suppresses the other applications on real networks has
not yet been quantified. The Iperf results can vary this way to
reflect the situation on the path at a particular time. The practical results
obtained by this method usually come in the range defined by “available
bandwidth” and the “real capacity” of the path. The Iperf
method uses brute force and
during a measurement sends to
the network hundreds of Mbytes of testing data.
is reason why it cannot be used for very frequent
believe that packet dispersion techniques
are quite comparable
to Iperf measurements. The packets dispersion methods used in ABwE or Pathchirp
are more modest in the sense that they don’t load more extra
monitoring traffic. The ABwE or Pathchirp
methods are also much less dependent on setting of parameters and so don’t
create a space for ambiguity (getting much different results) in one path.
However, these methods have other weak points, so the results could be
different in some situations (compare to the Iperf
or between themselves).
we are in the phase of evaluating ABwE and
One of our main tasks for the near future will be to prove that these tools
give good and accurate results in real
network s. We started this type of testing in the beginning of
2003 and the first stage was to find a relationship between the Iperf and
the ABwE results. Some results of this work are shown in the following graphs. The first
results were presented at PAM2003 in the talk by Jiri
Navratil (the paper “The Practical Approach to
Bandwidth Monitoring” contains only the results from 2002).
Figure 1: The result of the experiment for testing the narrow-band in the path between SLAC and NERSC. There was very low cross-traffic (XTraffic) on this path (red-line). The Bottleneck Capacity measured by ABwE (green line) agreed well with the narrow band capacity (100 Mbits/s). The estimate of an Available bandwidth is close to the capacity. The results of IEPM iperf measurements are black bars (repeated every 90 minutes). The agreement between ABW and Iperf is within 5%.
Figure 2: The results of the experiment between SLAC
and Internet2 office in
Figure 3: The results of the experiment between SLAC and RICE. There is an expected amount of cross-traffic (red-line) at about 30-40% capacity during the 24 hour period. The Bottleneck Capacity measured by ABwE (green line) shows an average value around 114 Mbits/s. The estimate of an Available bandwidth is moving between 60 - 100 Mbits/s. The agreement with the IEPM Iperf results (black bars) is very good (between 5-10%).
Figure 4: The results of the experiment for testing the ESnet high speed path between SLAC and FNAL. There is modest cross-traffic (red-line) at about 40 – 90 Mbits/s. The Bottleneck Capacity measured by ABwE (green line) shows an average value of 410 Mbits/s. The estimate of Available bandwidth varies between 300 - 400 Mbits/s with individual drops caused by randomly appearing cross traffic. The agreement with the IEPM Iperf results (black bars) is very good.
Figure 5: The results of the experiment for testing the high speed path between SLAC and NERSC. There is cross-traffic (red-line) with visibly increasing and decreasing trends over 24 hours. The Bottleneck Capacity measured by ABwE (green line) shows a value of 622 Mbits/s, which is the real capacity OC12 line between both labs. The estimate of the Available bandwidth is moving according to the cross-traffic profile. Individual drops in ABwE corresponds to the Iperf measurements made by IEPM-BW (in most cases). The drop is visible in the case when the measurements via Iperf (10 seconds) and the probing time of ABwE (0.5 seconds) match in the time. The agreement with the IEPM Iperf results (black bars) is very good, within 10%.
Figure 6: The results of the experiment for testing the high speed path between SLAC and CALTECH. There is high cross-traffic (red-line) at about 100 – 300 Mbits/s with a special pattern typical of paths where the total traffic is an aggregate of the activity of many people. The cross-traffic is increasing during the day time and decreasing during the night. The agreement with the IEPM Iperf results (black bars) is very good for most of the time. However, there are periods, when the Iperf gives much lower results. Compare to the previous examples, there is also different curve of the bottleneck capacity measured by ABwE (green line). This curve is also smoothly changing during the day. It is probably because the ABwE method is using relative relations between all measured values.
lightweight bandwidth estimation toolkit has been carefully evaluated and now provides
good bandwidth estimates in over 80% of the cases. It can make
an estimate in real time (< 1 second) with minimal impact (40kbits). The ABwE is providing feedback to IEPM. A
significant difference between measurements can be an indication that Iperf
used in IEPM needs to re-evaluate the parameters (windows & streams) used
for our heavier weight
estimator. A paper on ABwE was
presented at the PAM03 conference.
3. Future Accomplishments and Milestones
In previous paragraphs we have demonstrated the current
capability of ABwE. It allows us to do continuous monitoring with the
possibility of estimating available bandwidth on the path with an
Currently, we are trying to monitor more than 20 paths
to different destinations. We are probing several ESnet
sites, many 3 sites
on some sites we still have difficulties interpreting our results because they
did not match well with other measurements. It is hard to say which methods
give more accurate results in these situations. There are still more factors
which should be tested and verified. The problems with such situations can
possibly be split into three main categories:
devices on the path works with the different mode than
we expect (packet dispersion problems); There
is a traffic policy (traffic shaping) in some devices on the path which can
limit any type of transfer (including Iperf); And
the bottleneck node(s) problems on high speed segments, which can generate
bursts of packets.
the near future, we will concentrate on comparing both our methods (ABwE and
and developing them in frame of INCITE.
We are optimistic based on the first results obtained in March.
its real-time capability and low impact it is very suitable to use ABwE or
for providing real time feedback of anomalous changes in bandwidth performance.
We will work on the prediction
algorithms using ABwE as a source of information. Due to increasing interest
from the networking community to test ABwE methods, we are going to prepare a
standalone version for common use. We will also work on publishing our monitoring
results via tools used in other systems (as Mona lisa
used in Grid , etc.)
We will continue to discuss the possibility of how to run
chirping from many locations (main destination points as DOE Labs, CERN, IN2P3,
RAL, INFN and also sites located on the “paths” to these points) like PingER or
ABwE and permanently monitor selected
order to present the spectra of cross-traffic on these links on
our web pages. Current version of Pathchirp
report results into a file on remote site. In the version which could be useful
for wide area monitoring we will need to send this report on-line (as user
specified by option –w) into sender
site. Having this feature in pathchirp we can continue to discuss how to
present results from packet chirp tool on the SLAC and INCITE web pages and
other general usages of this tool.
The packet dispersion tools works on different time scales than most other standard methods, which are typically on the order of hours rather than msec. These tools can give much more information. Its greatest advantage and strongest feature will likely be “on-line mapping of cross-traffic” on a chosen path at a very fine time scale. Other tools are unable to do this because they would generate heavy load into the line to make such a measurement. This feature allows us to see details of traffic on the line.
We will conduct validation tests
of the new “sandwich probe” developed at RICE.
We will also continue programming new graphical representations (e.g.
zoom, subnet graphs, single route displays) to present the complicated tree
structures we obtain from the topo/tomo measurements.
We will include extending the node information available via drill-down, adding
new metrics (loss, CT) to how we display the
providing an archive of measurements with the ability to search historically.
We will continue deploy the traceroute measurements tools to more SciDAC
and Grid sites.
We will continue the development and implementation of the packet probing and tomo/topo toolsets.
· Regression test the new versions of the ABwE and chirp probing and topology/tomography tools, validate results, compare with other tools, find the regions of applicability
· Set up regular measurements with chosen tools, archive data, provide historical navigation of the archive.
· Provide public access to the archive via standard mechanisms.
· Modify an application to make a proof of concept of a network-aware application using the Rice measurements.
4. Project Management
We are planning a visit by Dr. J.Navratil to