Duties from PingER Supported by DOE MICS

IEPM Home Page IEPM Site Map Network Monitoring PingER IEPM Papers and Presentations

• Daily Pinger Analysis
• Installing a new PingER node
• Helpful Hints

Daily Pinger Analysis

Check status of Pinger Monitoring Host Data Collection

• The Hep site displays the health of daily Pinger data collection. This page is to be checked daily to assure that all of the data for the previous days have been acquired. When viewing the page, nodes that have green squares for the particular day you are checking are nodes for which all of the data has been acquired. Nodes with black squares indicate that only partial data has been taken for the node while nodes with red squares indicate that no data was taken. Unless there is a notable problem accessing the data from the remote sites, all data is expected to be acquired. To acquire the missing data:
  1. Note the nodename associated with the missing data.
  2. Log onto the pinger machine and cd to '/u6/analysis/hep'.

     jerrodw@pinger $ cd /afs/slac.stanford.edu/package/netmon/pinger

  3. Run getdata for the node with the missing data. This may take a few seconds or as much as a minute for some sites.

     jerrodw@pinger $ ./getdata.pl [nodename] [date (yyyy-mm-dd)]

  4. Once you have repeated the previous step for all the nodes with missing data, you can remake the graph by typing

     ./checkdata_gif.pl

     This will take about a minute to run.
  5. Check the webpage again to prove that all of the missing data has been acquired (all red and black squares are now green or black). If they are, you can skip the next two steps, otherwise continue.
  6. If .getdata.pl didn't get all of the data for the node, get the dataserver address for the particular node from our Nodedetails page and attepmpt to view the collected ping data.
  7. If the dataserver page is unavailable, this means that either the site is down or connection to its webserver is restricted. At this point, sending an email to the contact will be your best bet unless you have access to the monitoring host.
  8. If the web page does come up, try entering the parameters for yesterdays data. Sometimes the remote site has stopped taking data because the local disk is full or the cron job died. If this is the case you'll get something like 'the file contained no data', a similar error, or no data at all from your request.
  9. If a site goes down, you can also find the local administrator here (maintained by Maxim at FNAL). Especially if the site is one of:

     
     ANSP (cache01.ansp.br)
     Carleton University (netmon.physics.carleton.ca)
     CMU-HEP (cmuhep2.phys.cmu.edu)
     Daresbury (icfamon.dl.ac.uk)
     DESY (netmon.desy.de)
     DOE MICS (otf1.er.doe.gov)
     ESNET-NOC (netdb3.es.net)
     Fermilab (hepnrc.hep.net)
     ITEP (gate.itep.ru)
     KEK (yumj2.kek.jp)
     Niels Bohr Institute (www.nbi.dk)
     RMKI/KFKI (sgiserv.rmki.kfki.hu)
     Rutherford Lab (icfamon.rl.ac.uk)
     TRIUMF (sitka.triumf.ca)
     UMD-HEP (fermi.physics.umd.edu)
     

  Check Pinger Downsites for Remote Node Status

  1. The Pinger Downsites page displays a table of all Pinger remote sites that have been seen as down. From this page you can tell which Monitoring host has lost connectivity to the remote node and the number of days that the node has been down.
  2. Nodes that report a Pinger status of Up are nodes that have been down at some point in the past 7 days and are now able to be pinged. Sites that appear to be down should be reported to the network administrator or actions should be taken to have the node removed from the Monitoring host's list of available nodes to ping. The key here is identifying if the site is lost for good or has the chance to become pingable in the near future. This determines if you will have to delete the node from Nodedetails or if it should remain.
  3. Second in importance to the Pinger status column is the Ping status column which reports the current status of the remote sites.

  Installing a new PingER node

• New node entry is facilitated from the Nodedetails page.
• Write out the new database
  • The following steps have been automated and are executed via crontabs daily. If you need tosee the results of your entry immediately, you will have to execute the steps listed here:

    jerrodw@oceanus $ ./node.pl -o > /afs/slac.stanford.edu/package/netmon/pinger/nodes.cf

    jerrodw@oceanus $ /afs/slac/g/scs/net/pinger/bin/generateRSS-slaconly.pl

    jerrodw@oceanus $ /afs/slac/g/scs/net/pinger/bin/generateRSS.pl

• Send email to FNAL stating that the changes for the new node have been made and request that the web page is also updated(the web page is maintained by FNAL). In the email to Maxim, be sure to include the Fullname of the institution, Contact's name and email, dataserver path, traceroute path, and project type ONLY if it is a beacon site.
• Change the beacon list
  • Updates to the beacon's list are now also automated via the script write_offsitenodes stored in the /afs/slac/g/scs/net/pinger/bin directory. This script runs twice a day (1200 and 1800) from the trscrontab for the user 'pinger' on pinger.slac.stanford.edu.
  • Inform the local administrators of the new beacon node.
• After a few days look in the data to see who has not upgraded and send them a personal email. A day or two after that bug them some more. Some sites will upgrade immediately, some will eventually get to it if you prod them enough. Some will never do it.

Helpful Hints

1. Data from these tests are stored in /nfs/slac/g/net/pinger/pingerdata/hep/data/[nodename]/[file].
2. To view the data, use 'gzcat [file]'.
3. To edit, unzip the file but remember to zip it back.