UNIX Tools: pipe viewer (pv) – monitor the progress of data through a pipe

Pipe Viewer (pv) allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and estimated time of arrival.

pv can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

Install pv using the following command:

Debian and Ubuntu:

sudo apt-get install pv

CentOS, Fedora, RHEL:

yum install pv

FreeBSD:

make install clean -C /usr/ports/sysutils/pv

MacOS:

sudo port install pv

OpenSolaris:

pfexec pkg install pv

To use it, insert it in a pipeline between two processes, with the appropriate options. Its standard input will be passed through to its standard output and progress will be shown on standard error.

pv examples

pv will copy each supplied FILE in turn to standard output (- means standard input), or if no FILEs are specified just standard input is copied. This is the same behaviour as cat.

A simple example to watch how quickly a file is transferred using nc:

pv file | nc -w 1 hostdomain.com 7000

A similar example, transferring a file from another process and passing the expected size to pv:

cat file | pv -s 12345 | nc -w 1 hostdomain.com 7000

Use pv to view the progress while compressing files:

pv file | gzip > file.gz

You may stick several pv processes in between. For example, you can time how fast the data is being read from the disk and how much data is gzip outputting:

pv -cN source file | gzip | pv -cN gzip > file.gz

source: 120MB 0:00:15 [3.4MB/s] [=> ] 26% ETA 0:00:53
gzip: 6.5MB 0:00:15 [1.24MB/s] [ <=> ]

Here we specified the "-N" parameter to create a named stream. The "-c" parameter makes sure the output is not garbaged by one pv process writing over the other.

This example shows that "file" is being read at a speed of 3.4MB/s but gzip is writing data at only 1.24MB/s. We can immediately calculate the compression rate. It's 3.4/1.24!

Notice how the gzip does not include how much data is left or how fast it will finish. It's because the pv process after gzip has no idea how much data gzip will produce (it's just outputting compressed data from input stream).

Another similar example would be to pack the whole directory of files into a compressed tarball:

tar -czf - . | pv > out.tgz

In this example pv shows just the output rate of "tar -czf" command. Not very interesting and it does not provide information about how much data is left. We need to provide the total size of data we are tarring to pv, it's done this way:

tar -cf - . | pv -s $(du -sb . | awk '{print $1}') | gzip > out.tgz

What happens here is we tell tar to create "-c" an archive of all files in current directory "." (recursively) and output the data to stdout "-f -". Next we specify the size "-s" to pv of all files in current dir. The "du -sb . | awk '{print $1}'" returns number of bytes in current dir, and it gets fed as "-s" parameter to pv. Next we gzip the whole content and output the result to out.tgz file. This way "pv" knows how much data is still left to be processed and shows us an estimated time of arrival.

The manual of pipe viewer can be found here or typing

man pv

in console.