terminal

Linux commands for backups: pv

This post is about the Linux pipe viewer command, or pv. It's the second part of my beginners series on useful backup command-line tools for Linux.

Pipe viewer is a tool for monitoring the progress of data between Linux pipes. It's a powerful little command. It becomes more powerful when you combine it with other useful backup commands.

Let's start with a simple example. Copying files from one location to another:

$ cat /tmp/test > /home/rich/test
# or even:
$ cp /tmp/test /home/rich/test

Easy, but these lack progress bars. Instead, we can run it through pv:

$ pv /tmp/test > /home/rich/test
2.00GiB 0:00:01 [2.00GiB/s] [====>                          ] 19% ETA 0:00:04

We get the same action provided but with a handy progress bar. Sweet!

We can use it on a lot of commands, e.g. gzip:

$ gzip -c webserver.log | pv > webserver.log.gz

Going back to part 1, we can combine pv with tar to get the progress of a tar archive:

$ tar cf - --zstd myDir | pv > myDir.tar.zst

Note that we no longer get an ETA for this command. Pipe viewer doesn't know how big our directory is, unlike before with the file. We need to provide the size to pv using the -s argument:

$ tar cf - myDir | pv -s $(du -sb myDir | cut -f 1) | zstd > myDir.tar.zst

Here we are saying provide the size -s in bytes. We're then running a subcommand du, which estimates file space usage. Passing it the -s summarise, and -b bytes arguments. du outputs two columns, so we're simply using cut to get us the first column. The bytes.


Now we can combine it with ssh to send our backups over the wire with a progress bar:

$ tar cf - myDir | pv -s $(du -bs myDir | cut -f 1) | zstd | ssh user@remote-server "cat > myArchiveDir.tar.zst"

Ace! Now you've learnt pipe viewer, a useful tool to know! You can use it anytime you want to get progress between two commands. In part 3 we're going to introduce encryption to our backups.