Disk Throughputs Ajay Tirumala, Les Cottrell, Connie Logg, and I-Heng Mei

IEPM Home Page  IEPM Site Map  Network Monitoring  PingER  IEPM-BW Home IEPM Papers and Presentations

Questions

• What are the read and write disk throughputs achievable at various IEPM nodes?
• How can we accurately measure the sustainable throughput for large file transfers(without actually transferring a large file)?

Definitions

• cached, or plain write - writes do not perform a commit, i.e., the files are buffered and written to disk whenever the OS schedules it.
• commit each write - each write operation is committed to the disk
• commit at end - the file is explicitly committed to the disk after all writes
• block size - the size of each chunk of data read/written to the file system from our tests.

Results

• Read/write throughput at various IEPM nodes
• Block size v. write throughput
• File size v. write throughput
• Real File Transfers

Summary

Disk Reads

• In most cases (depending on the OS/file-system behaviour), the first read will necessitate a disk read and subsequent reads will be performed from the memory (cached copy).
• There are cases where the first read itself is from a (previous read/write's) cached copy. This will indicate a low paging activity or a large memory in the host system, which will enable the file to reside in the memory for the duration equal to the periodicity of the tests (about once a week).
• Some hosts do not cache the file at all.

Disk Writes

• In hosts where disk buffering is turned off (e.g. if close() also forces a commit), similar throughputs are obtained for 'plain' writes and 'committed' writes.
• In most cases, closing the file does not necessitate a commit to the disk before the close call returns. We use the fsync() call to commit the file
• The 'commit at end' option is most indicative of the disk write speeds achievable
• Using a large file size (~2GB) increased the throughputs only in a few cases

Block Sizes (effect on disk write throughput)

• Block size does not have a clear effect on 'commit at end' or 'cached writes' -- their throughputs generally remain fairly constant with different block sizes.
• For 'commit each write' mode, throughput generally increases as block size increases due to the relative seek time overhead being reduced as the amount of data committed each time becomes larger.
• 'Commit each write' throughput approaches the 'commit at end' throughput for file sizes of 4MB or greater on most nodes.

File Sizes (effect on disk write throughput)

• Throughput for 'commit at end' initially increases with file size, before leveling off.
• Throughput for 'cached writes' drop dramatically above a certain file size. This is due to the OS automatically committing the file to disk when the buffer cache becomes full.
• Both 'commit at end' and 'cached writes' approach the same throughput limit as the file size increases to infinity. This limit is the sustainable disk throughput for large transfers.
• Because 'commit at end' generally approaches the sustainble disk throughput much sooner (i.e. for smaller file sizes) than 'cached writes', then 'commit at end' can be used with small file transfers to estimate the sustainable disk throughput for a node. Note, this is only true for disk-only tests.

Real File Transfers (over the Internet)

How can we measure sustainable throughput for large file transfers over the Internet without transfering large amounts of data?

Most file transfer tools like BBFTP used 'cached writes'. We cannot use 'cached writes' with small file transfers to estimate sustainable throughput for large transfers. Since small files fit in the buffer cache, nothing will actually be committed to disk (at least immediately). So we would only be measuring BBFTP's network performance(and disk read throughput of the sender). While 'commit at end' can be used with small files to estimate sustainable disk-only throughput, this is not the case when the when the network is involved. Since the small file will fit the buffer cache, the entire file contents will be flushed only at the end of the transfer, thus forcing the network transfer and the actual disk writes to happen sequentially. For a large transfer, the disk activity will happen in parallel to the network transfer, since the buffer cache will fill up periodically, causing the OS to forcibly commit the data. Since the small transfer with 'commit at end' forces sequential network and disk activity instead of parallel, the throughput measurement will be lower than the actual sustainable throughput for BBFTP. A more appropriate method would be similar to 'commit each write', where the application explicitly commits the data periodically. For example, in order to estimate the performance of BBFTP for large files, we should transfer a small file using a variant of BBFTP that commits the data periodically. This method closely models the disk activity occurring in parallel with the network activity. The block size results suggest that we can commit the data every ~4 MB ('commit each write') to approximate the throughput of 'commit at end'. The file size results suggest we can transfer a relatively small file (32MB or 64MB) using 'commit at end' and still approximate the sustainable disk performance for large files. Thus, if we transfer a small file and commit the data every ~4MB, we should be able to approximate the performance of the application for large file transfers. We are currently trying out this theory with BBFTP. We are creating a modified version that commits the data periodically, and comparing the throughput of the modified BBFTP for small files versus the throughput of regular BBFTP for large files. We will add the results to this page.

To see the effect of different block sizes on disk write throughput, we varied the block size from 16KB to 64MB while keeping the file size constant at 128MB. From the results, we can see that varying the block size does not have a clear effect on either 'commit at end' or 'cached writes'. Block size does have a significant effect on the throughput in 'commit each write' mode, as the throughput is low for small block sizes and increases as block size increases, approaching the throughput of 'commit at end' after a certain point. This increase is likely due to a decrease in seek time overhead for the disk writes. When block size is small, 'commit each write' performs many small disk writes, resulting in a significant seek time relative to actual transfer time. As the block size is increased, this overhead is reduced and performance approaches that of 'commit at end'. For most nodes, a block size of 4MB causes 'commit each write' performance to be within 10% of 'commit at end' performance (see below).

    --- commit each write
    --- commit at end
    --- cached write

We varied the file size from 64KB to 1GB while keeping the block size constant at 64KB to measure the effect of file size on throughput. From the graphs below, we can see that file size does have an effect on both 'commit at end' and 'cached writes'. The 'commit at end' throughput generally increases with the throughput, probably due to the seek time becoming less significant as the committed data becomes larger. (Small files result in commiting a small amount of file data, thus the seek time is significant. As the file size increases, the transfer time dominates and the throughput begins to level off.) 'Cached write' throughputs are generally larger (due to OS buffering), but the throughput drops sharply above a certain file size threshold. This threshold represents the situation where the buffer cache fills up and the OS is forced to automatically commit the data to disk. Both 'commit at end' and 'cached writes' are subject to this threshold, and as the file size increases to infinity, their throughputs converge to the same limit - the sustainable disk throughput for large transfers. Clearly 'commit at end' approaches the limit much sooner, so we can use a smaller filesize to estimate sustainable disk throughput.(see below).

    --- commit each write
    --- commit at end
    --- cached write

Note: we assume 'max throughput' to be roughly equal to the 'commit at end' throughput for the largest file size in this test(1GB)