Linux window size observations Les Cottrell, SLAC, August 3, 2001 Linux 2.4 ========= We used bbcp on morpheus a Linux 2.4 host to send send a file with big windows to pharlap (Solaris 2.8). 3cottrell@morpheus:~>uname -a Linux morpheus 2.4.3-12 #1 Fri Jun 8 15:05:56 EDT 2001 i686 unknown The Ethernet interface on Morpheus was 10 Mbits/s 205cottrell@pharlap:~>uname -a SunOS pharlap 5.8 Generic_108528-03 sun4u sparc SUNW,Ultra-Enterprise Pharlap had a GE interface. The basic bbcp command for the attempts below was: /afs/slac/u/sf/abh/bbcp/bin/@sys/bbcp -f -v -b 4 -D -w 1024k .cshrc pharlap.slac.stanford.edu:/dev/null We also checked the results with Iperf. For an example of a trace and the Iperf commands see: http://www-iepm.slac.stanford.edu/monitoring/bulk/linux24-iperf-scale First attempt ------------- The wscale in the syn packet was 0. Second attempt -------------- Following http://www-didc.lbl.gov/tcp-wan.html#buffer_size we set echo "4096 87380 4194304" > /proc/sys/net/ipv4/tcp_rmem echo "4096 65536 4194304" > /proc/sys/net/ipv4/tcp_wmem The wscale option in the syn packet was set to 1. Third attempt ------------- We set echo 8388608 > /proc/sys/net/core/wmem_max echo 8388608 > /proc/sys/net/core/rmem_max The wscale option was set to 5, and the win was set to 5840. This corresponds to a net window size of: 5840*2^5 = 5840*32 = 186880 Bytes If we set the window in bbcp to 512k (-w 512k) then the wscale option on the syn packet was set to 4 and the window size to 5840, which corresponds to a net window size of 93440 Bytes. For a bbcp window size of 256k the wscale was 3, the window 5840, or a net window size of 46740 Bytes For a bbcp window size of 128k the wscale was 2, the window 5840 , or a net window size of 23360 Bytes For a bbcp window size of 64k the wscale was 1, the window 5840, or a net window size of 11680 Bytes For a bbcp window size of 32k the wscale was 1, the window 5840, or a net window size of 5840Bytes or in summary: BBCP window SYN wscale SYN window Net window 8192k 7 5840 747520 4096k 7 5840 747520 2048k 6 5840 373760 1024k 5 5840 186880 512k 4 5840 93440 256k 3 5840 46740 128k 2 5840 23360 64k 1 5840 11680 32k 0 5840 5840 However, see: http://www-iepm.slac.stanford.edu/monitoring/bulk/linux24-iperf-scale where it is observed that the Linux server receive window advertised during the transmission grows to 4MBytes. It turns out that Linux 2.4 no longer advertises its "maximum window size" in the SYN packet, instead the receiver keeps increasing the advertised window to just stay ahead of the transmitter. e.g., linux 2.4 receiver advertised window segment window (scaled bytes) 2 8672 3 11584 4 14464 5 17376 6 20256 7 23168 8 26048 9 28960 10 31840 ... 500 1571072 Linux 2.2 ========= We used bbcp on noric03 a Linux 2.2 host. 1cottrell@noric03:~>uname -a Linux noric03 2.2.16-3smp #1 SMP Mon Jun 19 19:00:35 EDT 2000 i686 unknown It had a 100Mbits/s fast Ethernet connetion 00:11.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone] We later confirmed the results using iperf on noric03. We used: more /proc/sys/net/core/wmem_max more /proc/sys/net/core/rmem_max more /proc/sys/net/core/rmem_default more /proc/sys/net/core/wmem_default and all these parameters were set to 65535. Sadly we were unable to change them. BBCP window SYN wscale SYN window Net window 1024k 0 32767 32767 512k 0 32767 32767 256k 0 32767 32767 128k 0 32767 32767 64k 0 32767 32767 32k 0 32120 32120 16k 0 16060 16060 With iperf we saw the following: 8cottrell@noric05:~>iperf -s -w 128k -p 5006 ------------------------------------------------------------ Server listening on TCP port 5006 TCP window size: 128 KByte (WARNING: requested 128 KByte) 1cottrell@pharlap:~>iperf -c noric05 -w 128k -p 5006 -t 1 ------------------------------------------------------------ Client connecting to noric05, TCP port 5006 TCP window size: 128 KByte ------------------------------------------------------------ [ 4] local 134.79.240.26 port 57773 connected with 134.79.125.205 port 5006 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 8.0 MBytes 63.3 Mbits/sec With snoop we identified that the SYN from noric03 advertised a receive window of 32767 and a wscale of 0. By the end of the transmission the receive window size advertized by noric03 was 63712. So though iperf reported using a 128KByte window it only grows to 64k (since the wscale is limited to 0). Linux 2.2 with big windows set ============================== We used iperf-1.2 on epaphus a Linux 2.2 host (where we could change the window/buffer settings). [cottrell@epaphus /]$ uname -a Linux epaphus 2.2.19 #2 Sat May 26 00:08:38 EDT 2001 i686 unknown We used the following settings: echo 8388608 /proc/sys/net/core/rmem_max echo 8388608 /proc/sys/net/core/wmem_max Iperf window SYN wscale SYN window Net window 1024k 5 32758 1048256 512k 5 16379 524128 256k 5 8166 261312 128k 5 4060 129920 64k 5 2007 64224 32k 5 1460 46720 16k 5 501 16032