Effective Techniques for Observing Linux Command Output Using 'Pv' Tools
Effective Techniques for Observing Linux Command Output Using ‘Pv’ Tools
Quick Links
- How to Install pv
- Using pv
- Copying a File with pv
- Copying Multiple Files with pv
- Using pv and tar to Create an Archive
- The pv Display Options
- Using pv With wc
- Installing the progress Command
- The Commands progress Works With
- Using Progress With Pipes
- Using Progress in Continual Monitor Mode
- 100% Completed
Instead of flying blind, use the Linux pv and progress commands to track a command’s progress. These utilities will give you progress bars for commands that don’t normally have any. You’ll see an estimated time until completion, too.
Starting a command from the terminal window can sometimes feel like a long-haul flight without a video screen. You have nothing to give any indication if all is well or if the process has hung, nor how close to completion it is. A flashing cursor isn’t very informative.
The pv and progress commands give you some statistics and a little visual feedback. You can see how close the process is to complete. That means you get an ETA for your running processes. Compared with staring at a cursor, that wins hands down.
How to Install pv
You must install pv.
To install pv on Ubuntu use this command:
sudo apt-get install pv
To install pv on Fedora use this command:
sudo dnf install pv
To install pv on Manjaro use this command:
sudo pacman -Syu pv
NeoDownloader - Fast and fully automatic image/video/music downloader.
Using pv
pv stands for pipe viewer . Piping has to be involved in the command somewhere. Here’s an example where we’re piping an ISO image through zip to make a compressed zip file of the ISO.
To slow the commands down enough that a screenshot could be taken, some of the files in the examples used for this article were stored on an old, slow, external USB called SILVERXHD.
pv /media/dave/SILVERXHD/gparted-live-1.0.0-1-amd64.iso | zip > gparted.zip
The information pv gives us can be seen in the bottom line of the display.
From left to right, the information that is displayed is:
- The data transferred so far.
- The time elapsed fo far.
- The data transfer rate (throughput).
- A progress bar and a percentage completed figure.
- The estimated time left before completion (ETA).
Copying a File with pv
To copy a file with output from pv, use this command:
pv /media/dave/SILVERXHD/gparted-live-1.0.0-1-amd64.iso > gparted.iso
We get a progress report as the file is copied.
Copying Multiple Files with pv
To copy multiple files and folders with pv we need to use a little trick. We use tar to move the files for us.
tar -c help-files/ | pv | tar -x -C Documents/
The tar -c help-files/ portion of the command instructs tar to create (-c) an archive of the files in the help-files folder. This is piped through pv so that we get a display of the progress. It is then piped back into tar for the last portion of the command. The archive is extracted (-x) and the directory is changed (-C) to Documents before the extraction.
So, the files and folders that are in help-files are copied to the Documents folder, with a progress display.
The output is slightly different this time.
We do not get an ETA. The progress bar now displays a moving indicator. It shows that the process is active, but it doesn’t grow from left to right like a traditional progress bar. pv is limited to displaying the information it can extract from the process that is being piped.
vMix 4K - Software based live production. vMix 4K includes everything in vMix HD plus 4K support, PTZ control, External/Fullscreen output, 4 Virtual Outputs, 1 Replay, 4 vMix Call, and 2 Recorders.Using pv and tar to Create an Archive
Copying files with pv and tar does not leave us with an archive file. A sort of “virtual” archive is created by tar, which is fed straight back into tar to extract the files. If our objective is to copy files, that is accomplished. But what if we want to create an archive file?
We can still use tar to create an archive file and get a progress report from pv. The options used with tar are -c (create archive), -z (compress with gzip) and -f (filename of the archive).
Note that we’re using - as the filename, which causes tar to use stdout , and to write its output to the terminal window. We don’t see that output because it is piped through pv.
The actual name of the archive is going to be the filename that we pipe the output from pv into. In this case, it is “help-files.tgz”.
tar -czf - ./help-files/ | pv > help-files.tgz
We get the same progress indicators as before, and the archive file is created for us.
The pv Display Options
There are a number of options you can use with pv to change the details of its report.
If you use any of these options, all of the other options are turned off. So if you want to have three of the display options in use, then you need to specify those three options.
Using pv without any options is the same as using the -pterb options.
- -p: display the percentage complete. This is the progress bar and the percentage completed figure.
- -t: display the elapsed time.
- -e: display the ETA.
- -r: display the rate of data transfer.
- -b: display the byte count (data transferred so far).
- -n: display the percentage as an integer. This prints the percentage completed as an integer figure, with each new update on a new line.
Let’s repeat the last command and pass the -p (percentage completed) option to pv.
tar -czf - ./help-files/ | pv - p > help-files.tgz
This turns off all of the other display options. pv provides the percentage completed element only.
Because pv doesn’t get a percentage completed figure from tar, the progress bar is replaced with a moving indicator. There is no percentage figure.
Using pv With wc
We can use pv to pipe a text file (or files) into wc. wc will then count the carriage returns, characters, and words and pv will give us a progress report.
Here we are piping all of the “.page” files in the help-files directory into wc.
When wc completes we can see our count of carriage returns (lines), characters and words from all of the “.page” files in the help-files folder.
Installing the progress Command
The progress command gives the same sort of useful information as pv, but it works with a specific set of Linux commands.
To install progress in Ubuntu, use this command:
sudo apt-get install progress
To install progress in Fedora, use this command:
sudo dnf install progress
To install progress in Manjaro, use this command:
sudo pacman -Syu progress
vMix Basic HD - Software based live production. vMix Basic HD includes 4 inputs, 3 cameras, streaming, recording, playlist.The Commands progress Works With
Typing progress in a terminal window and pressing Enter will give you a list of the commands that progress works with.
progress
Using Progress With Pipes
There are two techniques we can use to monitor commands with progress. The first is to use pipes.
The tar command is in the list of supported commands that progress can monitor, so let’s use tar.
The options we’ll use are the standard -c (create archive), -z (compress with gzip) and -f (filename) options. We’re going to create a compressed archive of everything in the help-files folder, and the archive will be named “help.tgz”.
We’re piping that into progress and using the -m (monitor) option so progress keeps reporting on the process until it has completed.
tar -czf help.tgz ./help-files/ | progress -m
The terminal window will show the progress of the tar command as it creates the archive.
Project Manager - Asset Browser for 3Ds Max
As each file is processed, it is listed, with the following information:
- The process ID.
- The process name.
- Percentage completed.
- Data processed and total size of the file.
- Data rate (throughput).
- Estimated time remaining (ETA).
You might be surprised to see a second data set appear. This first data set is for tar. The second one is for gzip. tar calls gzip to perform the compression. Because gzip is in the list of supported commands, progress reports on it.
Using Progress in Continual Monitor Mode
You can use progress in a real-time continual monitor mode by using the -M (monitor) option.
Type the following command in a terminal window:
progress -M
In another terminal window, type a command that is in the list of commands that progress can monitor.
We’re going to use cat. Commands that are over too quickly won’t register with progress, so we’ll list the contents of a very long text file.
cat words.page
In the terminal window with progress in it, you’ll see statistics for the cat command as it executes and works towards completion.
OtsAV Radio Webcaster When cat finishes listing the file progress returns to its waiting state.
Each time one of the commands it can report on performs a sizeable task, progress will automatically monitor it and report on it.
That’s pretty neat.
100% Completed
Take the guesswork out of wondering how a long-running command is doing, and take a break from contemplating your cursor with pv and progress .
| | Linux Commands | | |
| —————– | ———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————— | |
| Files | tar · pv · cat · tac · chmod · grep · diff · sed · ar · man · pushd · popd · fsck · testdisk · seq · fd · pandoc · cd · $PATH · awk · join · jq · fold · uniq · journalctl · tail · stat · ls · fstab · echo · less · chgrp · chown · rev · look · strings · type · rename · zip · unzip · mount · umount · install · fdisk · mkfs · rm · rmdir · rsync · df · gpg · vi · nano · mkdir · du · ln · patch · convert · rclone · shred · srm · scp · gzip · chattr · cut · find · umask · wc · tr | |
| Processes | alias · screen · top · nice · renice · progress · strace · systemd · tmux · chsh · history · at · batch · free · which · dmesg · chfn · usermod · ps · chroot · xargs · tty · pinky · lsof · vmstat · timeout · wall · yes · kill · sleep · sudo · su · time · groupadd · usermod · groups · lshw · shutdown · reboot · halt · poweroff · passwd · lscpu · crontab · date · bg · fg · pidof · nohup · pmap | |
| Networking | netstat · ping · traceroute · ip · ss · whois · fail2ban · bmon · dig · finger · nmap · ftp · curl · wget · who · whoami · w · iptables · ssh-keygen · ufw · arping · firewalld | |
- Title: Effective Techniques for Observing Linux Command Output Using 'Pv' Tools
- Author: Jeffrey
- Created at : 2024-08-30 16:48:46
- Updated at : 2024-08-31 16:48:46
- Link: https://eaxpv-info.techidaily.com/effective-techniques-for-observing-linux-command-output-using-pv-tools/
- License: This work is licensed under CC BY-NC-SA 4.0.