The best way to use pidstat is to pass a list of process ids with the -p parameter. The -u option reports CPU utilization. The -l parameter displays the process command name and its arguments. And we can optionally put time interval in seconds for the interval between reports and again, optionally, a count for number of reports.
$ pidstat -p 3947,3998 -l -u 2 3 Linux 3.0.0-14-generic (hostname) Wednesday 27 June 2012 _i686_ (2 CPU) 02:56:34 IST PID %usr %system %guest %CPU CPU Command 02:56:36 IST 3947 5.47 0.50 0.00 5.97 1 /usr/lib/firefox-11.0/firefox 02:56:36 IST 3998 0.00 1.00 0.00 1.00 0 /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja 02:56:36 IST PID %usr %system %guest %CPU CPU Command 02:56:38 IST 3947 3.50 0.50 0.00 4.00 0 /usr/lib/firefox-11.0/firefox 02:56:38 IST 3998 0.50 0.50 0.00 1.00 1 /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja 02:56:38 IST PID %usr %system %guest %CPU CPU Command 02:56:40 IST 3947 3.00 0.50 0.00 3.50 1 /usr/lib/firefox-11.0/firefox 02:56:40 IST 3998 0.50 1.00 0.00 1.50 0 /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja Average: PID %usr %system %guest %CPU CPU Command Average: 3947 3.99 0.50 0.00 4.49 - /usr/lib/firefox-11.0/firefox Average: 3998 0.33 0.83 0.00 1.16 - /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja
Instead of saying pidstat -p 3947,3998, we could have given the command as pidstat -C "firefox" to find processes with command name having the string "firefox". The string parameter can be a regular expression. But, while pidstat finds the processes with command name containing the string, it skips the processes with zero statistics values. Somehow, specifying the process ids seems to facilitate a more precise communication with the system.
The %usr column gives the percentage of CPU working in the user mode for the process. The %system column gives the percentage of CPU working in the kernel mode for the process. The %guest column gives percentage of the CPU time for the process running a virtual processor in a virtual machine. The %CPU column gives the total percentage of the CPU time. Had -I option been used, this would have been divided by the number of processors in the system. The CPU column tells which CPU has been assigned to the process. The last column gives the command and the parameters used.
The pidstat command is a part of the sysstat package for performance monitoring under Linux.
I/O Statistics
The -d option reports the I/O statistics for the concerned processes. For example,
$ pidstat -p 2093,2145 -l -I -d Linux 3.0.0-14-generic (hostname) Wednesday 27 June 2012 _i686_ (2 CPU) 09:27:47 IST PID kB_rd/s kB_wr/s kB_ccwr/s Command 09:27:47 IST 2093 14.77 35.66 8.37 /usr/lib/firefox-11.0/firefox 09:27:47 IST 2145 2.75 0.00 0.00 /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja
The kB_rd/s column gives the kilobytes per second read by the process. Similarly, kB_wr/s column gives the kilobytes per second written by the process. The next column, kB_ccwr/s gives the kilobytes/sec writing which has been canceled by this process. Suppose some other process has written to the pagecache, which has been overwritten by this process. Thus the earlier I/O need not be done now and has been canceled, thanks to this process.
Page faults and memory utilization
The -r option gives the page fault and memory utilization statistics. For example,
$ pidstat -p 1985,2232 -l -r Linux 3.0.0-14-generic (hostname) Thursday 28 June 2012 _i686_ (2 CPU) 04:58:20 IST PID minflt/s majflt/s VSZ RSS %MEM Command 04:58:20 IST 1985 159.51 0.61 643916 122204 12.73 /usr/lib/firefox-11.0/firefox 04:58:20 IST 2232 12.83 0.17 170216 19200 2.00 /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja
The minflt/s column gives the minor or soft page faults per second encountered by the process. Minor page faults occur when the concerned page is loaded in the memory but is not marked as loaded in the memory management unit as loaded. The minor page faults are not serious. The next column, majflt/s, gives the major page faults per second for the process. These require loading a memory page from the hard disk into the pagecache. A high value for major fault per second is definitely a cause for concern and the user should seriously think about increasing the main memory in the system. In the next column, VSZ, the virtual memory size for the process in kilobytes is given. And the next column, RSS, is the resident set size, the physical memory used by the process in kilobytes. Finally, the column, %MEM gives the physical memory used by the process as a percentage of the total available physical memory.
If while debugging a program, it is suspected to be having memory leaks, it can be checked using the pidstat command. For example,
$ pidstat -p 3245 -l -r 2 Linux 3.0.0-14-generic (hostname) Thursday 28 June 2012 _i686_ (2 CPU) 06:36:50 IST PID minflt/s majflt/s VSZ RSS %MEM Command 06:36:52 IST 3245 1.00 0.00 3804 244 0.03 ./buggy_program 06:36:54 IST 3245 1.00 0.00 4068 244 0.03 ./buggy_program 06:36:56 IST 3245 1.00 0.00 4332 244 0.03 ./buggy_program 06:36:58 IST 3245 1.00 0.00 4596 244 0.03 ./buggy_program 06:37:00 IST 3245 1.00 0.00 4860 244 0.03 ./buggy_program 06:37:02 IST 3245 1.00 0.00 5124 244 0.03 ./buggy_program 06:37:04 IST 3245 1.00 0.00 5388 244 0.03 ./buggy_program 06:37:06 IST 3245 1.00 0.00 5652 244 0.03 ./buggy_program 06:37:08 IST 3245 1.00 0.00 5916 244 0.03 ./buggy_program 06:37:10 IST 3245 1.00 0.00 6180 244 0.03 ./buggy_program 06:37:12 IST 3245 1.00 0.00 6444 244 0.03 ./buggy_program 06:37:14 IST 3245 1.00 0.00 6708 504 0.05 ./buggy_program 06:37:16 IST 3245 1.00 0.00 6972 504 0.05 ./buggy_program 06:37:18 IST 3245 1.00 0.00 7236 504 0.05 ./buggy_program ....
As the virtual memory size, VSZ, is increasing with time, the above process most likely has one or more memory leaks and the corresponding program should be checked for memory leaks.
Stack details
The -s option reports the stack utilization. The StkSize column gives the size of stack reserved for the process in kilobytes. All of it might not have been used. The StkRef column gives the amount of memory in kilobytes used as stack by the process. For example,
$ pidstat -p 2043,2194 -l -s Linux 3.0.0-14-generic (hostname) Thursday 28 June 2012 _i686_ (2 CPU) 12:16:25 IST PID StkSize StkRef Command 12:16:25 IST 2043 224 140 /usr/lib/firefox-11.0/firefox 12:16:25 IST 2194 136 40 /usr/lib/firefox-11.0/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox-11.0/omni.ja
Thread display
The statistics for threads are obtained by using the -t option. For example
$ pidstat -p 2043 -l -t Linux 3.0.0-14-generic (hostname) Thursday 28 June 2012 _i686_ (2 CPU) 12:22:53 IST TGID TID %usr %system %guest %CPU CPU Command 12:22:53 IST 2043 - 16.18 1.11 0.00 17.29 1 /usr/lib/firefox-11.0/firefox 12:22:53 IST - 2043 14.96 0.82 0.00 15.79 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2049 0.02 0.02 0.00 0.03 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2051 0.61 0.02 0.00 0.63 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2052 0.19 0.22 0.00 0.41 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2055 0.03 0.03 0.00 0.06 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2057 0.00 0.00 0.00 0.00 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2059 0.00 0.00 0.00 0.00 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2061 0.02 0.00 0.00 0.02 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2062 0.00 0.00 0.00 0.01 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2077 0.01 0.00 0.00 0.02 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2100 0.01 0.01 0.00 0.01 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2101 0.02 0.00 0.00 0.02 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2102 0.00 0.00 0.00 0.00 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2109 0.00 0.00 0.00 0.00 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2111 0.01 0.01 0.00 0.01 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2117 0.00 0.00 0.00 0.00 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2118 0.22 0.03 0.00 0.25 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2127 0.00 0.00 0.00 0.01 0 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2193 0.00 0.00 0.00 0.00 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 2269 0.00 0.00 0.00 0.00 1 |__/usr/lib/firefox-11.0/firefox 12:22:53 IST - 3682 0.00 0.00 0.00 0.00 0 |__/usr/lib/firefox-11.0/firefox
TGID and TID are the thread group id and thread id respectively.
Task switching
Task switching report is obtained by using the -w option. The cswch/s column gives the number of voluntary context switches per second. A voluntary context switch happens when the process or thread is not able to progress as it needs a resource which is not available and, hence, it relinquishes the processor. The nvcswch/s column gives the total number of involuntary context switches per second. An involuntary context switch happens when the process or thread is forced to relinquish the processor by the kernel as it has used the processor for its time slice. For example,
$ pidstat -p 1991 -w -l Linux 3.0.0-14-generic (hostname) Thursday 28 June 2012 _i686_ (2 CPU) 03:39:57 IST PID cswch/s nvcswch/s Command 03:39:57 IST 1991 96.15 17.14 /usr/lib/firefox-11.0/firefox
