IEPM

Setting up account on remote host

SLAC Home Page
Bulk throughput measurements | Bulk throughput simulation | Windows vs. streams | Effect of load on RTT and loss | Bulk file transfer measurements | QBSS measurements
Jerrod Williams and Les Cottrell, last update June 25 2002

Use ssh to logon to remote host

In this document we use "node2.cern.ch" as the example remote host, "cottrell" as an example of the account on the remote host, and pharlap.slac.stanford.edu as the monitoring host.
iepm@pharlap $ ssh node2.cern.ch -l cottrell
Warning: Permanently added 'node2.cern.ch,192.65.185.21' (RSA1) to the list of known hosts.
cottrell@node2.cern.ch's password: 

Copy the ssh public key

iepm@pharlap $ ssh cottrell@node2.cern.ch
Last login: Thu Dec 27 00:25:52 2001 from pharlap.slac.stanford.edu
[cottrell@node2 cottrell]$ ssh-keygen
Generating public/private rsa1 key pair.
Enter file in which to save the key (/home/cottrell/.ssh/identity): 
Created directory '/home/cottrell/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/cottrell/.ssh/identity.
Your public key has been saved in /home/cottrell/.ssh/identity.pub.
The key fingerprint is:
d1:25:04:57:04:b2:94:88:87:95:8b:ae:73:0e:13:ef cottrell@node2
  • When a new key-pair is needed for a specific machine:

    Get initial information about host

    In order to configure the account on the remote host and download the right software, it is first necessary to find out a couple of things about the account on the remote host. To do this you will need to execute the following 3 commands and note the results: N.b. if the userid of the account on the remote host is not "cottrell" then you will need to replace "cottrell" with the "userid" in the following commands:
    iepm@pharlap $ ssh cottrell@node2.cern.ch uname -a
    Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown
    iepm@pharlap $ ssh cottrell@node2.cern.ch pwd
    /home/cottrell
    iepm@pharlap $ ssh cottrell@node2.cern.ch which perl
    /usr/bin/perl
    iepm@pharlap $ 
    

    Edit aliases file

    Using the above information add the following to the aliases file. Look at the comments at the start of the aliases file for more information.
    $aliasname="node2.cern.ch"; #Alias for host for security/privacy reasons
    $PHONY{"$aliasname"} = "NODE2.CERN.CH"; #Real full name of host (must be in caps)
    $REMUSER{"$aliasname"} = "cottrell"; #Usually "cottrell"
    $HOMEDIR{"$aliasname"} = "/home/$REMUSER{$aliasname}"; #Home directory for remote user 
    $PERLPATH{"$aliasname"} = "#!/usr/bin/perl"; #Usually /usr/bin/perl or less common /usr/local/bin/perl
    $SSHOPTS{"$aliasname"} = ""; #Set to "-2" if using ssh version 2.
    

    Setup the remote files

    iepm@pharlap $ bin/remoteos.pl -h node2.cern.ch -D 1
    Wed Dec 26 09:33:15 2001 remoteos.pl starting for iepm on pharlap (134.79.240.26), choice=node2.cern.ch using:
       /afs/slac/package/netmon/bandwidth-tests/src/slaconly/aliases
    Skip plato.cacr.caltech.edu (from $PHONY{node1.cacr.caltech.edu}), does not match node2.cern.ch
    ...
    Skip morticia.utdallas.edu (from $PHONY{node1.utdallas.edu}), does not match node2.cern.ch
    Executing ssh   cottrell@node2.cern.ch mkdir /home/cottrell/bin /home/cottrell/pinger /home/cottrell/pinger/load
    Executing ssh   cottrell@node2.cern.ch mkdir /home/cottrell/package
    Executing ssh   cottrell@node2.cern.ch mkdir /home/cottrell/iperf-1.2
    Host node2.cern.ch initialized
    Executing ssh  -l cottrell -f node2.cern.ch uname -a
    Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown
    29/-1 OS=Linux, Ver=2.4.7-10, Name=node2, sys=i386_linux24, pwd=/home/cottrell, perl=/usr/bin/perl for node2.cern.ch
    Will 
    Copying bin/arch.pl to node2.cern.ch with perl path=/usr/bin/perl
    scp  -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/arch.pl
    Copying bin/iperf.pl to node2.cern.ch with perl path=/usr/bin/perl
    scp  -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/iperf.pl
    Copying bin/startiperf.pl to node2.cern.ch with perl path=/usr/bin/perl
    scp  -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/startiperf.pl
    Copying bin/tcpflow.pl to node2.cern.ch with perl path=/usr/bin/perl
    scp  -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/tcpflow.pl
    Copying bin/tcpload.pl to node2.cern.ch with perl path=/usr/bin/perl
    scp  -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/tcpload.pl
    Will ssh  cottrell@node2.cern.ch mv /home/cottrell/bin/bbcp /home/cottrell/bin/bbcp-old
    mv: cannot stat `/home/cottrell/bin/bbcp': No such file or directory
    Will scp  -p /afs/slac/package/bbcp/bin/i386_linux24/bbcp cottrell@node2.cern.ch:/home/cottrell/bin/bbcp
    Will ssh  cottrell@node2.cern.ch mv /home/cottrell/bin/bbftp /home/cottrell/bin/bbftp-old
    mv: cannot stat `/home/cottrell/bin/bbftp': No such file or directory
    Will scp  -p /afs/slac.stanford.edu/u/sf/iepm/bin/bbftp/bin/i386_linux24/bbftp cottrell@node2.cern.ch:/home/cottrell/bin/bbftp
    Will ssh  cottrell@node2.cern.ch mv /home/cottrell/bin/bbftpd /home/cottrell/bin/bbftpd-old
    mv: cannot stat `/home/cottrell/bin/bbftpd': No such file or directory
    Will scp  -p /afs/slac.stanford.edu/u/sf/iepm/bin/bbftp/bin/i386_linux24/bbftpd cottrell@node2.cern.ch:/home/cottrell/bin/bbftpd
    Will scp  -p /afs/slac.stanford.edu/package/scsutils/bin.i386_linux24/iperf cottrell@node2.cern.ch:/package/iperf-1.2/iperf
    scp: /package/iperf-1.2/iperf: No such file or directory
    Skip 128.55.27.64 (from $PHONY{node2.nersc.gov}), does not match node2.cern.ch
    Skip iperf2.nslabs.ufl.edu (from $PHONY{node2.nslabs.ufl.edu}), does not match node2.cern.ch
    Wed Dec 26 09:33:52 2001 remoteos.pl done for iepm on pharlap
    
    Pay attention to the above output ofr any files that were not copied to the remote host. All files not copied via "remoteos" requires a localmake in order to be copied. Click here to find instructions for conducting a localmake on remote hosts.

    Selecting the optimum windows and streams

    Use ssh to execute the bin/arch.pl on the account on the remote host, to check that the TCP window sizes are properly set (see http://www-iepm.slac.stanford.edu/monitoring/bulk/#os). If not you will need to send email to the remote host contact to request that she/he set the parameters.
    iepm@pharlap $ ssh cottrell@node2.cern.ch bin/arch.pl
    Thu Dec 27 03:00:37 2001 arch.pl findings for  (...)
    Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown
    ...
    cat /proc/sys/net/core/wmem_max = 8388607
    ;cat /proc/sys/net/core/rmem_max = 8388607
    ;cat /proc/sys/net/core/rmem_default = 65535
    ;cat /proc/sys/net/core/wmem_default = 65535
    ;cat /proc/sys/net/ipv4/tcp_rmem = 4096 87380   8388607
    ;cat /proc/sys/net/ipv4/tcp_wmem = 4096 65530   8388607
    ;ls: /afs/slac.stanford.edu/: No such file or directory
    cat /proc/sys/net/core/wmem_max cat /proc/sys/net/core/rmem_max cat /proc/sys/net/core/rmem_default cat /proc/sys/net/core/wmem_default cat /proc/sys/net/ipv4/tcp_rmem cat /proc/sys/net/ipv4/tcp_wmem
    iepm@pharlap $ 
    
    Then use ssh to run bin/iperf.pl, this will start iperf servers on the remote host ports 5001 through 5009 with window sizes of 8KBytes, 16KBytes, 32KBytes, 55KBytes, 64KBytes, 128KBytes, 256KBytes, 512KBytes and 1024Kbytes respectively:
    iepm@pharlap $ ssh -f cottrell@node2.cern.ch bin/iperf.pl
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 32.0 KByte (WARNING: requested 16.0 KByte)
    ------------------------------------------------------------
    ...
    Server listening on TCP port 5009
    TCP window size:  2.0 MByte (WARNING: requested  1.0 MByte)
    ------------------------------------------------------------
    
    (see
    Problems for what to do if you do not get the above output).

    From the monitoring host run bin/tcpload,pl to measure the throughput for multiple windows sizes and numbers of streams:

    iepm@pharlap $ bin/tcpload.pl -h node2.cern.ch -l userid -D 1 -W 1 -M 20000 -m 1000 -P "1,20,8,12,120,4,25,60,40,90,2,30" > tmp/tcpload-node2.cern.ch-yyyymmdd.csv
    Wed Dec 26 15:42:10 2001 tcpload.pl (1/1) on SunOS measuring flow rates with /afs/slac.stanford.edu/u/sf/iepm/bin/tcpflow.pl for 
                    8k 1024k 16k 512k 32k 256k 128k 64k windows (8) and 
                    1 20 5 12 8 10 4 25 11 3 7 40 15 9 2 30 6 streams (17)
                    32k50031024k500916k5002512k50088k500164k500555k5004128k5006256k50072048k50104096k5011 ports
                    with 100 byte ping packets, 10 tests/case 
                    from pharlap(134.79.240.26) to node2.cern.ch
    Wed Dec 26 15:42:37 2001 tcpload.pl(1/1) 1(1)/17 1(8/8k)/8 13s, got   2.90Mbps in 0.02u 0.09s   1.05% 10.48, for 10 unloaded 100 byte pings received 10 with  160/167/187,
    Wed Dec 26 15:43:04 2001 tcpload.pl(1/1) 1(1)/17 2(1024/1024k)/8 40s, got  24.30Mbps in 0.04u 0.42s   4.35% 10.57, for 10 unloaded 100 byte pings received 10 with  160/160/160,
    ...
    
    In this example we disabled the Web100 processing with the -W option (to conserve memory), and only make a measurement if the product of windows and streams is in the range 1000-20000 KBytes. The file created (/tmp/tcpload-node2-20011226 in the above example) is in comma separated format suitable for importing into Excel. Import it into Excel and plot throughput vs. streams for varying window sizes. Select the optimum windows and streams settings, i.e. the setting that gives close to the maximum throughput for the minimum streams*window product.

    Then run kill-bbcp.pl on the monitoring host to kill the iperf processes previously started by iperf.pl:


    iepm@pharlap $ bin/kill-bbcp.pl -h node2.cern.ch -p siperf -x 1 -D 3
    Wed Dec 26 22:35:45 2001 kill-bbcp.pl for siperf starting for iepm on pharlap (134.79.240.26), choice=node2.cern.ch
    kill-bbcp.pl: OS=SunOS, for 21/32 possible hosts
    Skip plato.cacr.caltech.edu or node1.cacr.caltech.edu, does not match node2.cern.ch
    ...
    Executing ssh -l cottrell -f node2.cern.ch uname -a
    Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown
    Executing ssh -l cottrell -f node2.cern.ch echo $PATH
    cottrell@node2.cern.ch: OS=Linux, Ver=2.4.7-10, Name=node2, sys=i386_linux24, pwd=, perl= for node2.cern.ch
    AFS cell=, path=/u/sf/iepm/bin.sun5:/u/sf/iepm/bin:/usr/local/bin:/usr/afsws/bin:/usr/afsws/etc:/usr/apps:/usr/ccs/bin:/bin:/usr/bin:/etc:/usr/etc:/usr/sbin:/usr/ucb:/usr/dt/bin:/usr/openwin/bin:/usr/local/X11R5/bin:.
    ssh -l cottrell -f node2.cern.ch ps -efl | grep cottrell | grep '/siperf'
    24 siperf processes for cottrell on node2.cern.ch
    000 S cottrell 17508     1  0  69   0    -  2531 rt_sig 07:11 pts/1    00:00:00 /home/cottrell/bin/siperf -s -w 16k -p 5002
     040 S cottrell 17509 17508  0  69   0    -  2531 do_pol 07:11 pts/1    00:00:00 /home/cottrell/bin/siperf -s -w 16k -p 5002
    ...
     040 S cottrell 17538 17534  0  69   0    -  2531 wait_f 07:11 pts/1    00:00:00 /home/cottrell/bin/siperf -s -w 512k -p 5008
    1(24): ssh -l cottrell -f node2.cern.ch kill 17508 for:
      000 S cottrell 17508     1  0  69   0    -  2531 rt_sig 07:11 pts/1    00:00:00 /home/cottrell/bin/siperf -s -w 16k -p 5002
    bash: kill: (17509) - No such pid
    2(24): ssh -l cottrell -f node2.cern.ch kill 17509 for:
      040 S cottrell 17509 17508  0  69   0    -  2531 do_pol 07:11 pts/1    00:00:00 /home/cottrell/bin/siperf -s -w 16k -p 5002
    3(24): ssh -l cottrell -f node2.cern.ch kill 17511 for:
      000 S cottrell 17511     1  0  69   0    -  2531 rt_sig 07:11 pts/1    00:00:00 /home/cottrell/bin/siperf -s -w 32k -p 5003
    ...
    0 siperf process still on node2.cern.ch
    Skip 128.55.27.64 or node2.nersc.gov, does not match node2.cern.ch
    Skip iperf2.nslabs.ufl.edu or node2.nslabs.ufl.edu, does not match node2.cern.ch

    Finish editing aliases on monitoring host

    $REMHOST{"$aliasname"}= "OS=Linux 2.4&CPU=1496 MHz&NIC=1000 Mbps";

    Use bin/arch.pl on remote host to find OS version and cpu's speed and number of cpus.
    iepm@pharlap $ ssh cottrell@node2.cern.ch bin/arch.pl
    Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown
    Path: 
    Home directory: /home/cottrell
    Perl path: perl: /usr/bin/perl /usr/share/man/man1/perl.1.gz
    ls: /afs/slac.stanford.edu/: No such file or directory
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 15
    model           : 0
    model name      : Intel(R) Pentium(R) 4 CPU 1500MHz
    stepping        : 7
    cpu MHz         : 1495.484
    cache size      : 256 KB
    fdiv_bug        : no
    hlt_bug         : no
    f00f_bug        : no
    coma_bug        : no
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 2
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm
    bogomips        : 2981.88
    
    more /proc/sys/net/core/wmem_max more /proc/sys/net/core/rmem_max more /proc/sys/net/core/rmem_default more /proc/sys/net/core/wmem_default more /proc/sys/net/ipv4/tcp_rmem more /proc/sys/net/ipv4/tcp_wmem
    more /proc/sys/net/core/wmem_max = 8388607
    ;more /proc/sys/net/core/rmem_max = 8388607
    ;more /proc/sys/net/core/rmem_default = 65535
    ;more /proc/sys/net/core/wmem_default = 65535
    ;more /proc/sys/net/ipv4/tcp_rmem = 4096        87380   8388607
    ;more /proc/sys/net/ipv4/tcp_wmem = 4096        65530   8388607
    

    Finish editing aliases file on monitoring host

    See the comments in the file for the meanings of the parameters that you need to add for the new host.

    Test with run-bw-tests

    On the monitoring host execute:
    iepm@pharlap $ bin/run-bw-tests -n node2.cern.ch
    
    Typical successful output*.

    Problems

    bin/iperf.pl gives:
    /home/cottrell/bin/siperf: error while loading shared libraries: libstdc++-libc6.1-1.so.2: cannot open shared object file: No such file or directory
    
    There is an incompatibility between the library used to create the iperf binary at SLAC and the libraries at the remote host. You will have to make iperf at the remote host, and set the following configuration variable in the aliases file:
    $IPERFVER{"aliasname"}= "localmake";
    where aliasname is the alias for the remote host, e.g. node2.cern.ch in the above examples. You can download iperf from the
    Iperf site.
    Click here to find instructions for conducting a localmake on remote hosts.
    * Page not accessible from non SLAC host.