VALVE: Copy Data, Limiting Transfer Rate

by John Walker

This page describes, in Unix manual page style, a program available for downloading from this site which copies data while enforcing a specified maximum transfer rate in bytes per second by pausing between data blocks. This permits bulk data transfers without “locking up” disc or network bandwidth to the detriment of other applications and/or users. It can be thought of as the equivalent of the Unix nice command for I/O.

NAME

valve - copy data, enforcing transfer rate limitation

SYNOPSIS

valve [ options ] [ infile [ outfile ] ]

DESCRIPTION

Usually, when you copy one file to another or transfer data between applications, you want things to finish as soon as possible—why wait any longer than necessary? However, when the operation in question consumes scarce shared disc or network transfer bandwidth (for example, when copying large files between machines on a local network or across the Internet), it may be wise to limit the transfer rate to avoid degrading performance for other users or applications. In an ideal world, wise and benevolent operating systems would automatically throttle bandwidth to achieve this goal, but as we all know, ours is not an ideal world, nor are operating systems, for the most part, wise or benevolent.

valve is a Unix/Linux/etc. command line utility which copies an input source to an output destination, pausing as necessary between blocks of data to achieve a requested mean transfer rate. Options permit measuring transfer rate without enforcing a limit and determining the timing granularity of the system on which the program is run.

OPTIONS

Long (--) options may be abbreviated to their shortest unique prefix and single-letter options without arguments may be aggregated. Arguments to single-letter options may immediately follow the option or be separated by white space.

--alwayswait, -a
If the measured mean transfer rate has fallen sufficiently behind the requested rate such that no pause between blocks is indicated, valve normally skips the pause. If this option is set, a minimum-length pause will always occur after each block copied. This option can avoid pathological behaviour on copies with an inconsistent transfer rate.
--blocksize, -b size
Data will be copied in blocks of size bytes. The size argument may contain commas and include a size factor suffix as described below. If no --blocksize is specified, a block size will be chosen such that four blocks per second are copied at the requested transfer --rate, with an upper limit on block size of one megabyte.
--copyright
Print program copyright information.
--help, -u
Print how-to-call information.
--measure, -m
Measure, but do not limit, the transfer rate and print it at the end of the copy. Note that the --blocksize setting may influence the transfer rate.
--rate, -r bytesec
The transfer rate will be limited to bytesec bytes per second. The argument may contain commas and include a size factor suffix as described below. If no --rate is specified, a rate of one megabyte per second will be used.
--summary, -s
Print a summary of the number of bytes copied and the actual transfer rate at the end of the copy.
--timingtest secs
Run a timing test for secs seconds to determine the minimum wait interval supported by the operating system on the machine running the test. Knowing this value permits skipping pauses which would inadvertently reduce the transfer rate below the desired value. (See the --alwayswait option.) The timing test shows the default minimum wait time built into valve and the measured minimum interval, both in nanoseconds. If there is a radical difference between these values (there's no need to worry about a factor of three or less), you may wish to specify the actual value with the --waitmin option or rebuild the program, configuring it with the MIN_SLEEP_NSEC= variable set to the measured value.
--version
Print program version information.
--waitmin nsecs
Pauses shorter than nsecs nanoseconds will be skipped unless the --alwayswait option is specified. The waitmin argument may contain commas and include a size factor suffix as described below. If no --waitmin is specified, the value built into the program with the MIN_SLEEP_NSEC= configuration variable will be used, with a default of “10,000,000” (10 milliseconds).

SCALE FACTOR SUFFIXES

Arguments to the --blocksize, --rate, and --waitmin options can include commas to separate groups of digits, and may be followed by a suffix indicating a scale factor for the numeric argument as follows. Note that these suffix specifications are case-sensitive.

c character 1
w word 2
b block 512
KB, kB   kilo 1,000
K, k kibi 1,024
MB mega 1,000,000
M mebi 1,048,576
GB giga 1,000,000,000
G gibi 1,073,741,824
TB tera 1,000,000,000,000
T tebi 1,099,511,627,776
PB peta 1,000,000,000,000,000
P pebi 1,125,899,906,842,624
EB exa 1,000,000,000,000,000,000
E exbi 1,152,921,504,606,846,976
ZB zetta 1,000,000,000,000,000,000,000
Z zebi 1,180,591,620,717,411,303,424
YB yotta 1,000,000,000,000,000,000,000,000
Y yobi   1,208,925,819,614,629,174,706,176

A trailing “D” may be used to denote a decimal power of ten instead of “B”.

EXIT STATUS

valve returns status 0 if processing was completed without errors, 1 if an I/O or internal error occurred, and 2 if processing could not be performed at all due, for example, to a nonexistent input file or command line syntax error.

EXAMPLES

Suppose you're logged into a local system and you with to transfer the contents of the current directory to a directory named “/mirror/files/” on a remote system named somewhere.ratburger.net, while limiting the transfer rate to 256 Kb/sec to avoid saturating your outbound bandwidth or the inbound bandwidth of the destination site. You can accomplish this with: 

    tar cf - . | valve --rate 256K | \
        ssh somewhere.ratburger.net "(cd /mirror/files; tar xf -)"

FILES

If no infile is specified or infile is a single “-”, valve reads from standard input; if no outfile is given, or outfile is a single “-”, output is sent to standard output. The input and output are processed strictly serially; consequently valve may (and usually will) be used in pipelines. The program can process files of any size supported by the system with any binary content whatsoever.

BUGS

This is a Unix application through and through, and will require substantial work to port to other operating systems, particularly in the area of high-precision waits. It does, however, work on 32-bit Windows systems with Cygwin installed.

valve uses the nanosleep function, if available, to pause between blocks of data. If the system does not support that function, usleep is used, if available, and failing that, an emulation of usleep with setitimer and signal. If none of these facilities are available, you're out of luck.

It is tacky using “B” to denote a decimal power of ten in scale factor suffixes, but that's how dd does it, so I opted in favour of compatibility. You can use a trailing “D”, if you prefer, to indicate a decimal power.

Please report bugs and documentation errors to Bravo Uniform Golf Sierra @ Foxtrot Oscar Uniform Romeo Mike India Lima Alpha Bravo Decimal Charlie Hotel. If you don't have any idea what I just said, please consult this document.

SEE ALSO

cat(1), dd(1), nanosleep(2), setitimer(2), signal(2), usleep(3)

Download

*   valve C source code: valve-1.0.tar.gz
*   Read valve source code (requires Acrobat Reader)

AUTHOR

John Walker
http://www.fourmilab.ch/

This software is in the public domain. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, without any conditions or restrictions. This software is provided “as is” without express or implied warranty.