Iperf server on pharlap ======================= 149cottrell@pharlap:~>uname -a SunOS pharlap 5.8 Generic_108528-03 sun4u sparc SUNW,Ultra-Enterprise Pharlap has a GE interface. 127cottrell@pharlap:~>------------------------------------------------------------ Server listening on TCP port 5011 TCP window size: 4.0 MByte ------------------------------------------------------------ [ 5] local 134.79.240.26 port 5011 connected with 134.79.24.165 port 1246 [ ID] Interval Transfer Bandwidth [ 5] 0.0- 6.6 sec 6.3 MBytes 7.6 Mbits/sec [ 4] local 134.79.240.26 port 5011 connected with 134.79.24.165 port 1247 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 6.6 sec 6.3 MBytes 7.6 Mbits/sec Run snoop on pharlap ======================= 150cottrell@pharlap:~>sudo snoop -o /tmp/snoop morpheus.slac.stanford.edu pharlap.slac.stanford.edu AFS Password: Using device /dev/ge (promiscuous mode) 0 Run Iperf client on morpheus ============================ 23cottrell@morpheus:~/package/iperf-1.2>uname -a Linux morpheus 2.4.3-12 #1 Fri Jun 8 15:05:56 EDT 2001 i686 unknown Morpheus has 10 Mbits/s Ethernet interface 24cottrell@morpheus:~/package/iperf-1.2>echo "4096 87380 4194304" > /proc/sys/net/ipv4/tcp_rmem 25cottrell@morpheus:~/package/iperf-1.2>echo "4096 65536 4194304" > /proc/sys/net/ipv4/tcp_wmem 26cottrell@morpheus:~/package/iperf-1.2>echo 8388608 > /proc/sys/net/core/wmem_max 27cottrell@morpheus:~/package/iperf-1.2>echo 8388608 > /proc/sys/net/core/rmem_max 28cottrell@morpheus:~/package/iperf-1.2>./iperf -c pharlap -w 4096k -t 1 -p 5011 ------------------------------------------------------------ Client connecting to pharlap, TCP port 5011 TCP window size: 8.0 MByte (WARNING: requested 4.0 MByte) ------------------------------------------------------------ [ 3] local 134.79.24.165 port 1247 connected with 134.79.240.26 port 5011 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 6.7 sec 6.3 MBytes 7.9 Mbits/sec Capture, format and look at packets from snoop on pharlap ========================================================= 151cottrell@pharlap:~>snoop -i /tmp/snoop > ! /tmp/temp 152cottrell@pharlap:~>head -20 /tmp/temp 161cottrell@pharlap:~>head -20 /tmp/temp 1 0.00000 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=22 S=57751 Ack=854238578 Seq=816845766 Len=20 Win=24820 2 0.00340 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57751 S=22 Ack=816845786 Seq=854238578 Len=60 Win=8576 3 0.00464 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=22 S=57751 Ack=854238638 Seq=816845786 Len=0 Win=24820 4 2.30046 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=22 S=57751 Ack=854238638 Seq=816845786 Len=20 Win=24820 5 0.00503 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57751 S=22 Ack=816845806 Seq=854238638 Len=20 Win=8576 6 0.00396 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=5011 S=1249 Syn Seq=3229447493 Len=0 Win=5840 Options= 7 0.00004 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=1249 S=5011 Syn Ack=3229447494 Seq=1369293131 Len=0 Win=32772 Options= 8 0.00003 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=22 S=57751 Ack=854238658 Seq=816845806 Len=0 Win=24820 9 0.00044 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=5011 S=1249 Ack=1369293132 Seq=3229447494 Len=0 Win=45 Options= 10 0.00106 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57751 S=22 Ack=816845806 Seq=854238658 Len=324 Win=8576 11 0.00133 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=5011 S=1249 Ack=1369293132 Seq=3229447494 Len=1448 Win=45 Options= 12 0.00103 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=1249 S=5011 Ack=3229448942 Seq=1369293132 Len=0 Win=32772 Options= 13 0.00018 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=5011 S=1249 Ack=1369293132 Seq=3229448942 Len=1448 Win=45 Options= 14 0.00008 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=1249 S=5011 Ack=3229450390 Seq=1369293132 Len=0 Win=32772 Options= 15 0.00112 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=5011 S=1249 Ack=1369293132 Seq=3229450390 Len=1448 Win=45 Options= 16 0.00146 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=5011 S=1249 Ack=1369293132 Seq=3229451838 Len=1448 Win=45 Options= 17 0.00019 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=1249 S=5011 Ack=3229453286 Seq=1369293132 Len=0 Win=32772 Options= 18 0.00103 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=5011 S=1249 Ack=1369293132 Seq=3229453286 Len=1448 Win=45 Options= 19 0.00127 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=5011 S=1249 Ack=1369293132 Seq=3229454734 Len=1448 Win=45 Options= 20 0.00012 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=1249 S=5011 Ack=3229456182 Seq=1369293132 Len=0 Win=32772 Options= Conclusion despite the Iperf client on Morpheus saying it has given a window of 8MByte (twice what was requested), the snoop output (line 6) shows morpheus has requested a window of 5840*2^wscale, and wscale is 7, i.e a window of 747520 Bytes. In fact looking at all the packets from morpheus to port 5011 on pharlap (e.g. line 11) they all have a window of 45 Bytes which after applying the scaling factor is 5760 Bytes. This is probably OK since it is the receive window that has to accept the the data, and morpheus is acting as the client sending the bulk data and so is mainly accepting ACKs. Repeat the above but with morpheus as the Iperf client and pharlap as the server ================================================================================ ================================================================================ Start iperf server on morpheus: =============================== 6cottrell@morpheus:~/package/iperf-1.2>./iperf -s -w 4096k -p 5011 ------------------------------------------------------------ Server listening on TCP port 5011 TCP window size: 8.0 MByte (WARNING: requested 4.0 MByte) ------------------------------------------------------------ [ 6] local 134.79.24.165 port 5011 connected with 134.79.240.26 port 57754 [ ID] Interval Transfer Bandwidth [ 6] 0.0- 6.4 sec 4.1 MBytes 5.3 Mbits/sec Start snoop on pharlap ====================== 165cottrell@pharlap:~>sudo snoop -o /tmp/snoop morpheus.slac.stanford.edu pharlap.slac.stanford.edu Using device /dev/ge (promiscuous mode) 6200 ^C Start iperf client on pharlap ============================= 129cottrell@pharlap:~>iperf -c morpheus -w 4096k -p 5011 -t 1 ------------------------------------------------------------ Client connecting to morpheus, TCP port 5011 TCP window size: 4.0 MByte ------------------------------------------------------------ [ 4] local 134.79.240.26 port 57754 connected with 134.79.24.165 port 5011 ^C[ ID] Interval Transfer Bandwidth [ 4] 0.0- 6.4 sec 4.1 MBytes 5.1 Mbits/sec Capture, format and look at captured packets ============================================ 66cottrell@pharlap:~>snoop -i /tmp/snoop > ! /tmp/temp 167cottrell@pharlap:~>head -20 /tmp/temp 1 0.00000 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Syn Seq=1555618278 Len=0 Win=32781 Options= 2 0.00137 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Syn Ack=1555618279 Seq=4046972169 Len=0 Win=5792 Options= 3 0.00004 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555618279 Len=0 Win=32783 Options= 4 0.00176 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555618279 Len=1448 Win=32783 Options= 5 0.00004 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555619727 Len=1448 Win=32783 Options= 6 0.00003 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555621175 Len=1448 Win=32783 Options= 7 0.00304 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1555619727 Seq=4046972170 Len=0 Win=67 Options= 8 0.00003 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555622623 Len=1448 Win=32783 Options= 9 0.00003 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555624071 Len=1448 Win=32783 Options= 10 0.00001 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1555621175 Seq=4046972170 Len=0 Win=90 Options= 11 0.00015 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555625519 Len=952 Win=32783 Options= 12 0.00002 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555626471 Len=1448 Win=32783 Options= 13 0.00581 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1555622623 Seq=4046972170 Len=0 Win=113 Options= 14 0.00003 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555627919 Len=1448 Win=32783 Options= 15 0.00002 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1555624071 Seq=4046972170 Len=0 Win=135 Options= 16 0.00002 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555629367 Len=1448 Win=32783 Options= 17 0.00011 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1555625519 Seq=4046972170 Len=0 Win=158 Options= 18 0.00002 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555630815 Len=1448 Win=32783 Options= 19 0.00004 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972170 Seq=1555632263 Len=1448 Win=32783 Options= 20 0.00007 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57751 S=22 Ack=816846566 Seq=854240122 Len=92 Win=8576 Pharlap (Solaris) offers a window of 32781*2^7 = 4194304 (see line 1) Morpheus (Linux) offers a window of 5792 * 2^7 = 741376 (see line 2) However, if one looks at the window sizes advertised by Morpheus one finds they increase starting at 67 (after scaling this = 8576 Bytes) on line 7, but growing to 31403 (after scaling 4019584) by the end, this can be seen below (for example see line 6197): 172cottrell@pharlap:~>tail /tmp/temp 6191 0.00009 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1559892103 Seq=4046972170 Len=0 Win=31290 Options= 6192 0.00012 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1559893551 Seq=4046972170 Len=0 Win=31313 Options= 6193 0.00002 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1559894503 Seq=4046972170 Len=0 Win=31313 Options= 6194 0.00009 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1559895951 Seq=4046972170 Len=0 Win=31335 Options= 6195 0.00002 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1559897399 Seq=4046972170 Len=0 Win=31358 Options= 6196 0.00009 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1559898847 Seq=4046972170 Len=0 Win=31380 Options= 6197 0.00013 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Ack=1559900296 Seq=4046972170 Len=0 Win=31403 Options= 6198 0.00009 MORPHEUS.SLAC.Stanford.EDU -> PHARLAP.SLAC.Stanford.EDU TCP D=57754 S=5011 Fin Ack=1559900296 Seq=4046972170 Len=0 Win=31403 Options= 6199 0.00001 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=5011 S=57754 Ack=4046972171 Seq=1559900296 Len=0 Win=32783 Options= 6200 0.00109 PHARLAP.SLAC.Stanford.EDU -> MORPHEUS.SLAC.Stanford.EDU TCP D=22 S=57751 Ack=854240330 Seq=816846566 Len=0 Win=24820 So maybe all is well with Morpheus and Linux 2.4, at least when it is receiving the bulk data it sets the receiver end window OK. As for the client/sender it is only receiving ACKs and so does not need to advertize a large window.