![]() |
Setting up account on remote hostBulk throughput measurements | Bulk throughput simulation | Windows vs. streams | Effect of load on RTT and loss | Bulk file transfer measurements | QBSS measurements Jerrod Williams and Les Cottrell, last update June 25 2002 |
iepm@pharlap $ ssh node2.cern.ch -l cottrell Warning: Permanently added 'node2.cern.ch,192.65.185.21' (RSA1) to the list of known hosts. cottrell@node2.cern.ch's password:
iepm@pharlap $ ssh cottrell@node2.cern.ch Last login: Thu Dec 27 00:25:52 2001 from pharlap.slac.stanford.edu [cottrell@node2 cottrell]$ ssh-keygen Generating public/private rsa1 key pair. Enter file in which to save the key (/home/cottrell/.ssh/identity): Created directory '/home/cottrell/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/cottrell/.ssh/identity. Your public key has been saved in /home/cottrell/.ssh/identity.pub. The key fingerprint is: d1:25:04:57:04:b2:94:88:87:95:8b:ae:73:0e:13:ef cottrell@node2
You must specify a key type (-t). Usage: ssh-keygen [options] Options: bits Number of bits in the key to create. -c Change comment in private and public key files. -e Convert OpenSSH to IETF SECSH key file. -f filename Filename of the key file. -i Convert IETF SECSH to OpenSSH key file. -l Show fingerprint of key file. -p Change passphrase of private key file. -q Quiet. -y Read private key file and print public key. -t type Specify type of key to create. -B Show bubblebabble digest of key file. -C comment Provide new comment. -N phrase Provide new passphrase. -P phrase Provide old passphrase.*run 'ssh-keygen with the -t option and the type as 'rsa' and continue to the next step.
[cottrell@node2 cottrell]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/rich/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/rich/.ssh/id_rsa. Your public key has been saved in /home/rich/.ssh/id_rsa.pub. The key fingerprint is: 15:80:5b:66:ae:42:4b:97:6f:45:c8:e7:d7:d6:88:17 [cottrell@node2 cottrell]$ ls -la total 28 drwx------ 3 cottrell other 4096 Dec 26 17:54 . drwxr-xr-x 12 root root 4096 Dec 17 12:54 .. -rw-r--r-- 1 cottrell other 24 Dec 14 00:40 .bash_logout -rw-r--r-- 1 cottrell other 191 Dec 14 00:40 .bash_profile -rw-r--r-- 1 cottrell other 124 Dec 14 00:40 .bashrc -rw-r--r-- 1 cottrell other 3511 Dec 14 00:40 .screenrc drwx------ 2 cottrell other 4096 Dec 26 17:54 .ssh [cottrell@node2 cottrell]$ ls .ssh identity identity.pub [cottrell@node2 cottrell]$ scp iepm@pharlap.slac.stanford.edu:.ssh/identity.pub .ssh/authorized_keys The authenticity of host 'pharlap.slac.stanford.edu (134.79.240.26)' can't be established. DSA key fingerprint is 8a:92:0c:b2:fe:08:29:69:c9:82:19:7a:9f:3a:60:65. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'pharlap.slac.stanford.edu,134.79.240.26' (DSA) to the list of known hosts. iepm@pharlap.slac.stanford.edu's password: identity.pub 100% |*******************| 331 00:00 [cottrell@node2 cottrell]$ logout
debug1: Remote protocol version 2.0, remote software version 3.1.2 SSH Secure Shell (non-commercial)as opposed to say:
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.2.3p1The OpenSSH key format appears with each public dsa key being in a single line prefaced by "ssh-dss ", e.g.
>23iepm@pharlap:~>cat .ssh/id_dsa.pub ssh-dss AAAAB3NzaC1kc3MAAACBALUpFtrYqDtiQgB7YAh0N/kLHLs1BDzeUl8JZT/3OwFOi9VlTjA <snip>This has to be reformatted into a multiline SSH Communications, Inc. format:
72iepm@pharlap:~>cat .ssh/id_dsa_reformat.pub ---- BEGIN SSH2 PUBLIC KEY ---- Subject: root Comment: "1024-bit dsa hostkey" AAAAB3NzaC1kc3MAAACBAOVADGoXtH3LwWjZG88xASRoNvygAufXNlAfByJzhZLk4daksi jvkixzwkfQP1cUgxmIfYFj3S4NEW4qcWnaYBGS1ISmfqkg2xbhUxumKPVOH2b/RDp7Skzs 0hOw6EoqS35a0z3JTMBjprRYf82AJyptQo7dBcX+YExrs9Na48s3AAAAFQDe1coC6N/wWA 9ZDG/eolHsrZb52QAAAIAeGh9Ony0aUncR1C6ev0dH5gAISB24DcG8qsGqKZspo0IwoiPh u/FqfyjAT14TivXhemF7EQuRzPDhP8Mic1SvHHXV0tyJ59kzBtdn7/F6LlMb4rFXzyRMGX 2/Gu9Cp25OG/8rIWCnaqgaz0zjxynKxq3XxSwSBQpj/R1tLlaN2wAAAIBf4aEXyphTSF0T G7F1RcX0QezyLl4J54gQLZt6EvJdoPL7mw/zYmzBqbb5yIKGgzUuhqMMf6GW2UBcQBRraS 4hTwbN+0tl82yHzszkjHEPjYG8uhGULCJlQLjMtNgACi+ewktJGnCPt7/4+75GsPS0HVkn Bu08RuqKV5iuvfABAw== ---- END SSH2 PUBLIC KEY ----This reformatted public key then has to be copied into the apropriate directory on the remote host (use the man ssh command on the remote host to see where to put the file, sometimes it is in the .ssh2 directory in .ssh2/authorized_keys2 and a pointer may need to be put into .ssh2/authorization which is of the form:
[cottrell@node2 ~]$ cat .ssh2/authorization Key authorized_keys2
21iepm@antonia:~>ssh -v cottrell@node2.cern.ch OpenSSH_3.2.2p1-SLAC, SSH protocols 1.5/2.0, OpenSSL 0x0090603fThen you need to change the mode settings on the .ssh file on the remotet host (in this case node2.cern.ch), for example:debug1: Trying RSA authentication with key '/u/sf/iepm/.ssh/identity' debug1: Remote: Authentication refused: bad ownership or modes for directory /home/cottrell/.ssh debug1: Server refused our key.
[cottrell@node2 ~]$ ls -ld .ssh drwxrwxr-x 2 cottrell cottrell 4096 Jun 24 21:02 .ssh [cottrell@node2 ~]$ chmod g-rwx .ssh [cottrell@node2 ~]$ ls -ld .ssh drwx---r-x 2 cottrell cottrell 4096 Jun 24 21:02 .ssh [cottrell@node2 ~]$ chmod o-rx .ssh [cottrell@node2 ~]$ ls -ld .ssh drwx------ 2 cottrell cottrell 4096 Jun 24 21:02 .ssh
[cottrell@node2 cottrell]$ ssh-keygen -t rsa -f [homedir]/.ssh/id_rsa_[uniquename] Generating public/private rsa key pair. Enter file in which to save the key (/home/rich/.ssh/id_rsa_[uniquename]): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/rich/.ssh/id_rsa_[uniquename]. Your public key has been saved in /home/rich/.ssh/id_rsa_[uniquename].pub. The key fingerprint is: 15:80:5b:66:ae:42:4b:97:6f:45:c8:e7:d7:d6:88:17
Host [hostname] IdentityFile /u/sf/iepm/.ssh/id_rsa_[uniquename]
iepm@pharlap $ ssh cottrell@node2.cern.ch uname -a Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown iepm@pharlap $ ssh cottrell@node2.cern.ch pwd /home/cottrell iepm@pharlap $ ssh cottrell@node2.cern.ch which perl /usr/bin/perl iepm@pharlap $
$aliasname="node2.cern.ch"; #Alias for host for security/privacy reasons
$PHONY{"$aliasname"} = "NODE2.CERN.CH"; #Real full name of host (must be in caps)
$REMUSER{"$aliasname"} = "cottrell"; #Usually "cottrell"
$HOMEDIR{"$aliasname"} = "/home/$REMUSER{$aliasname}"; #Home directory for remote user
$PERLPATH{"$aliasname"} = "#!/usr/bin/perl"; #Usually /usr/bin/perl or less common /usr/local/bin/perl
$SSHOPTS{"$aliasname"} = ""; #Set to "-2" if using ssh version 2.
iepm@pharlap $ bin/remoteos.pl -h node2.cern.ch -D 1
Wed Dec 26 09:33:15 2001 remoteos.pl starting for iepm on pharlap (134.79.240.26), choice=node2.cern.ch using:
/afs/slac/package/netmon/bandwidth-tests/src/slaconly/aliases
Skip plato.cacr.caltech.edu (from $PHONY{node1.cacr.caltech.edu}), does not match node2.cern.ch
...
Skip morticia.utdallas.edu (from $PHONY{node1.utdallas.edu}), does not match node2.cern.ch
Executing ssh cottrell@node2.cern.ch mkdir /home/cottrell/bin /home/cottrell/pinger /home/cottrell/pinger/load
Executing ssh cottrell@node2.cern.ch mkdir /home/cottrell/package
Executing ssh cottrell@node2.cern.ch mkdir /home/cottrell/iperf-1.2
Host node2.cern.ch initialized
Executing ssh -l cottrell -f node2.cern.ch uname -a
Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown
29/-1 OS=Linux, Ver=2.4.7-10, Name=node2, sys=i386_linux24, pwd=/home/cottrell, perl=/usr/bin/perl for node2.cern.ch
Will
Copying bin/arch.pl to node2.cern.ch with perl path=/usr/bin/perl
scp -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/arch.pl
Copying bin/iperf.pl to node2.cern.ch with perl path=/usr/bin/perl
scp -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/iperf.pl
Copying bin/startiperf.pl to node2.cern.ch with perl path=/usr/bin/perl
scp -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/startiperf.pl
Copying bin/tcpflow.pl to node2.cern.ch with perl path=/usr/bin/perl
scp -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/tcpflow.pl
Copying bin/tcpload.pl to node2.cern.ch with perl path=/usr/bin/perl
scp -p /tmp/remoteos.pl cottrell@node2.cern.ch:/home/cottrell/bin/tcpload.pl
Will ssh cottrell@node2.cern.ch mv /home/cottrell/bin/bbcp /home/cottrell/bin/bbcp-old
mv: cannot stat `/home/cottrell/bin/bbcp': No such file or directory
Will scp -p /afs/slac/package/bbcp/bin/i386_linux24/bbcp cottrell@node2.cern.ch:/home/cottrell/bin/bbcp
Will ssh cottrell@node2.cern.ch mv /home/cottrell/bin/bbftp /home/cottrell/bin/bbftp-old
mv: cannot stat `/home/cottrell/bin/bbftp': No such file or directory
Will scp -p /afs/slac.stanford.edu/u/sf/iepm/bin/bbftp/bin/i386_linux24/bbftp cottrell@node2.cern.ch:/home/cottrell/bin/bbftp
Will ssh cottrell@node2.cern.ch mv /home/cottrell/bin/bbftpd /home/cottrell/bin/bbftpd-old
mv: cannot stat `/home/cottrell/bin/bbftpd': No such file or directory
Will scp -p /afs/slac.stanford.edu/u/sf/iepm/bin/bbftp/bin/i386_linux24/bbftpd cottrell@node2.cern.ch:/home/cottrell/bin/bbftpd
Will scp -p /afs/slac.stanford.edu/package/scsutils/bin.i386_linux24/iperf cottrell@node2.cern.ch:/package/iperf-1.2/iperf
scp: /package/iperf-1.2/iperf: No such file or directory
Skip 128.55.27.64 (from $PHONY{node2.nersc.gov}), does not match node2.cern.ch
Skip iperf2.nslabs.ufl.edu (from $PHONY{node2.nslabs.ufl.edu}), does not match node2.cern.ch
Wed Dec 26 09:33:52 2001 remoteos.pl done for iepm on pharlap
Pay attention to the above output ofr any files that were not copied to the remote host. All files
not copied via "remoteos" requires a localmake in order to be copied. Click here to find instructions for
conducting a localmake on remote hosts.
scp: FATAL: Executing ssh1 in compatibility mode failed (Check that scp1 is in your PATH). lost connectionYou will need to send email to the administrator at the remote host to request them to install OpenSSH, see FAQ.
iepm@pharlap $ ssh cottrell@node2.cern.ch bin/arch.pl Thu Dec 27 03:00:37 2001 arch.pl findings for (...) Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown ... cat /proc/sys/net/core/wmem_max = 8388607 ;cat /proc/sys/net/core/rmem_max = 8388607 ;cat /proc/sys/net/core/rmem_default = 65535 ;cat /proc/sys/net/core/wmem_default = 65535 ;cat /proc/sys/net/ipv4/tcp_rmem = 4096 87380 8388607 ;cat /proc/sys/net/ipv4/tcp_wmem = 4096 65530 8388607 ;ls: /afs/slac.stanford.edu/: No such file or directory cat /proc/sys/net/core/wmem_max cat /proc/sys/net/core/rmem_max cat /proc/sys/net/core/rmem_default cat /proc/sys/net/core/wmem_default cat /proc/sys/net/ipv4/tcp_rmem cat /proc/sys/net/ipv4/tcp_wmem iepm@pharlap $Then use ssh to run bin/iperf.pl, this will start iperf servers on the remote host ports 5001 through 5009 with window sizes of 8KBytes, 16KBytes, 32KBytes, 55KBytes, 64KBytes, 128KBytes, 256KBytes, 512KBytes and 1024Kbytes respectively:
iepm@pharlap $ ssh -f cottrell@node2.cern.ch bin/iperf.pl ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 32.0 KByte (WARNING: requested 16.0 KByte) ------------------------------------------------------------ ... Server listening on TCP port 5009 TCP window size: 2.0 MByte (WARNING: requested 1.0 MByte) ------------------------------------------------------------(see Problems for what to do if you do not get the above output).
From the monitoring host run bin/tcpload,pl to measure the throughput for multiple windows sizes and numbers of streams:
iepm@pharlap $ bin/tcpload.pl -h node2.cern.ch -l userid -D 1 -W 1 -M 20000 -m 1000 -P "1,20,8,12,120,4,25,60,40,90,2,30" > tmp/tcpload-node2.cern.ch-yyyymmdd.csv
Wed Dec 26 15:42:10 2001 tcpload.pl (1/1) on SunOS measuring flow rates with /afs/slac.stanford.edu/u/sf/iepm/bin/tcpflow.pl for
8k 1024k 16k 512k 32k 256k 128k 64k windows (8) and
1 20 5 12 8 10 4 25 11 3 7 40 15 9 2 30 6 streams (17)
32k50031024k500916k5002512k50088k500164k500555k5004128k5006256k50072048k50104096k5011 ports
with 100 byte ping packets, 10 tests/case
from pharlap(134.79.240.26) to node2.cern.ch
Wed Dec 26 15:42:37 2001 tcpload.pl(1/1) 1(1)/17 1(8/8k)/8 13s, got 2.90Mbps in 0.02u 0.09s 1.05% 10.48, for 10 unloaded 100 byte pings received 10 with 160/167/187,
Wed Dec 26 15:43:04 2001 tcpload.pl(1/1) 1(1)/17 2(1024/1024k)/8 40s, got 24.30Mbps in 0.04u 0.42s 4.35% 10.57, for 10 unloaded 100 byte pings received 10 with 160/160/160,
...
In this example we disabled the Web100 processing with the -W option (to conserve memory), and
only make a measurement if the product of windows and streams is in the range 1000-20000 KBytes.
The file created (/tmp/tcpload-node2-20011226 in the above example) is in
comma separated format suitable for importing into Excel. Import it into Excel and plot
throughput vs. streams for varying window sizes. Select the optimum windows and
streams settings, i.e. the setting that gives close to the maximum throughput
for the minimum streams*window product.
Then run kill-bbcp.pl on the monitoring host to kill the iperf processes previously started by iperf.pl:
iepm@pharlap $ bin/kill-bbcp.pl -h node2.cern.ch -p siperf -x 1 -D 3
Wed Dec 26 22:35:45 2001 kill-bbcp.pl for siperf starting for iepm on pharlap
(134.79.240.26), choice=node2.cern.ch
kill-bbcp.pl: OS=SunOS, for 21/32 possible hosts
Skip plato.cacr.caltech.edu or node1.cacr.caltech.edu, does not match
node2.cern.ch
...
Executing ssh -l cottrell -f node2.cern.ch uname -a
Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown
Executing ssh -l cottrell -f node2.cern.ch echo $PATH
cottrell@node2.cern.ch: OS=Linux, Ver=2.4.7-10, Name=node2, sys=i386_linux24,
pwd=, perl= for node2.cern.ch
AFS cell=, path=/u/sf/iepm/bin.sun5:/u/sf/iepm/bin:/usr/local/bin:/usr/afsws/bin:/usr/afsws/etc:/usr/apps:/usr/ccs/bin:/bin:/usr/bin:/etc:/usr/etc:/usr/sbin:/usr/ucb:/usr/dt/bin:/usr/openwin/bin:/usr/local/X11R5/bin:.
ssh -l cottrell -f node2.cern.ch ps -efl | grep cottrell | grep '/siperf'
24 siperf processes for cottrell on node2.cern.ch
000 S cottrell 17508 1 0 69
0 - 2531 rt_sig 07:11 pts/1 00:00:00
/home/cottrell/bin/siperf -s -w 16k -p 5002
040 S cottrell 17509 17508 0 69
0 - 2531 do_pol 07:11 pts/1 00:00:00
/home/cottrell/bin/siperf -s -w 16k -p 5002
...
040 S cottrell 17538 17534 0 69
0 - 2531 wait_f 07:11 pts/1 00:00:00
/home/cottrell/bin/siperf -s -w 512k -p 5008
1(24): ssh -l cottrell -f node2.cern.ch kill 17508 for:
000 S cottrell 17508 1 0
69 0 - 2531 rt_sig 07:11
pts/1 00:00:00 /home/cottrell/bin/siperf -s -w 16k -p 5002
bash: kill: (17509) - No such pid
2(24): ssh -l cottrell -f node2.cern.ch kill 17509 for:
040 S cottrell 17509 17508 0 69
0 - 2531 do_pol 07:11 pts/1 00:00:00
/home/cottrell/bin/siperf -s -w 16k -p 5002
3(24): ssh -l cottrell -f node2.cern.ch kill 17511 for:
000 S cottrell 17511 1 0
69 0 - 2531 rt_sig 07:11
pts/1 00:00:00 /home/cottrell/bin/siperf -s -w 32k -p 5003
...
0 siperf process still on node2.cern.ch
Skip 128.55.27.64 or node2.nersc.gov, does not match node2.cern.ch
Skip iperf2.nslabs.ufl.edu or node2.nslabs.ufl.edu, does not match
node2.cern.ch
iepm@pharlap $ ssh cottrell@node2.cern.ch bin/arch.pl Linux node2 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001 i686 unknown Path: Home directory: /home/cottrell Perl path: perl: /usr/bin/perl /usr/share/man/man1/perl.1.gz ls: /afs/slac.stanford.edu/: No such file or directory processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 0 model name : Intel(R) Pentium(R) 4 CPU 1500MHz stepping : 7 cpu MHz : 1495.484 cache size : 256 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm bogomips : 2981.88 more /proc/sys/net/core/wmem_max more /proc/sys/net/core/rmem_max more /proc/sys/net/core/rmem_default more /proc/sys/net/core/wmem_default more /proc/sys/net/ipv4/tcp_rmem more /proc/sys/net/ipv4/tcp_wmem more /proc/sys/net/core/wmem_max = 8388607 ;more /proc/sys/net/core/rmem_max = 8388607 ;more /proc/sys/net/core/rmem_default = 65535 ;more /proc/sys/net/core/wmem_default = 65535 ;more /proc/sys/net/ipv4/tcp_rmem = 4096 87380 8388607 ;more /proc/sys/net/ipv4/tcp_wmem = 4096 65530 8388607
iepm@pharlap $ bin/run-bw-tests -n node2.cern.chTypical successful output*.
/home/cottrell/bin/siperf: error while loading shared libraries: libstdc++-libc6.1-1.so.2: cannot open shared object file: No such file or directory