The sar command under Linux and Unix systems gives the system activity reports. As the system works, a record of activities is kept in certain counters in the kernel. The sar command gives statistics regarding the CPU, I/O, paging, devices, memory, swap space, network, run queue length and load average, interrupts and power management. At the very basic level, it gives the CPU utilization. For example,
$ sar 2 4 Linux 3.0.0-14-generic (hostname) Sunday 15 July 2012 _i686_ (2 CPU) 06:04:43 IST CPU %user %nice %system %iowait %steal %idle 06:04:45 IST all 7.29 0.00 3.02 0.00 0.00 89.70 06:04:47 IST all 9.73 0.00 3.74 2.00 0.00 84.54 06:04:49 IST all 6.25 0.00 1.25 0.00 0.00 92.50 06:04:51 IST all 12.72 0.00 2.99 0.00 0.00 84.29 Average: all 9.00 0.00 2.75 0.50 0.00 87.75
The above command gives the CPU utilization at an interval of 2 seconds 4 (count) times. If we skip the count, the reports are given continuously. If we skip both the interval and the count, the sar command gives the CPU utilization for the current day.
SYSSTAT
sar is a part of the sysstat package for performance monitoring under Linux. The data collection for generating the reports is done by the sadc program. The first step is to enable the data collection in /etc/default/sysstat.
# # Default settings for /etc/init.d/sysstat, /etc/cron.d/sysstat # and /etc/cron.daily/sysstat files # # Should sadc collect system activity informations? Valid values # are "true" and "false". Please do not put other values, they # will be overwritten by debconf! ENABLED="true"
After changing the ENABLED flag in /etc/default/sysstat, the data collection is started by the command,
$ sudo invoke-rc.d sysstat start
sadc is called from the shell script, /usr/lib/sysstat/sa1, which in turn is started from the init script /etc/init.d/sysstat. The data is recorded in a binary file, /var/log/sysstat/sadd, where dd is the day of the month for the current day. This way, the data is recorded for each day of the month. Thus, using the relevant data files, sar gives a system administrator the facility to check the various system activity reports for the system's operation in the past.
File options
Using the -o option, sar writes the output data into a file, in addition to the display on the screen. Using the -f option, this file can be read later on get the reports on the screen.
$ sar 2 4 -o sar-data Linux 3.0.0-14-generic (hostname) Monday 16 July 2012 _i686_ (2 CPU) 10:47:45 IST CPU %user %nice %system %iowait %steal %idle 10:47:47 IST all 6.25 0.00 2.25 0.00 0.00 91.50 10:47:49 IST all 6.02 0.00 2.26 0.00 0.00 91.73 10:47:51 IST all 6.73 0.00 1.50 0.00 0.00 91.77 10:47:53 IST all 6.28 0.00 2.26 0.00 0.00 91.46 Average: all 6.32 0.00 2.07 0.00 0.00 91.61 $ file sar-data sar-data: data $ sar -f sar-data Linux 3.0.0-14-generic (hostname) Monday 16 July 2012 _i686_ (2 CPU) 10:47:45 IST CPU %user %nice %system %iowait %steal %idle 10:47:47 IST all 6.25 0.00 2.25 0.00 0.00 91.50 10:47:49 IST all 6.02 0.00 2.26 0.00 0.00 91.73 10:47:51 IST all 6.73 0.00 1.50 0.00 0.00 91.77 10:47:53 IST all 6.28 0.00 2.26 0.00 0.00 91.46 Average: all 6.32 0.00 2.07 0.00 0.00 91.61
By using -e [ hh:mm:ss ] in conjunction with -o or -f, it is possible to set an end time for a report. For example,
$ sar -f ./sa14 -e 13:00:00 Linux 3.0.0-14-generic (hostname) Saturday 14 July 2012 _i686_ (2 CPU) 12:00:01 IST CPU %user %nice %system %iowait %steal %idle 12:05:01 IST all 3.82 0.00 0.96 0.17 0.00 95.05 12:15:01 IST all 3.94 0.00 1.03 1.84 0.00 93.18 12:25:01 IST all 3.83 0.00 0.95 0.09 0.00 95.12 12:35:01 IST all 3.53 0.00 0.90 0.03 0.00 95.54 ... 12:25:02 IST all 5.98 0.00 1.49 5.43 0.00 87.10 12:35:01 IST all 5.96 0.00 1.60 3.49 0.00 88.96 12:45:01 IST all 5.58 0.00 1.39 1.92 0.00 91.11 12:55:01 IST all 5.96 0.00 1.48 1.81 0.00 90.76 Average: all 7.17 0.21 1.84 2.36 0.00 88.43
gives a report till 01:00 PM. If -e option is given without specifying any time, a default of 18:00:00 is taken and the report is given till 06:00 PM. Similarly, while reading the sar reports from a file with the -f option, the option -s [hh:mm:ss] can be used to set the start time of the report.
sardd reports
The sa2 command is a script which is called by the script, /etc/cron.daily/sysstat. The latter is scheduled by cron or anacron. sa2 writes daily reports in files named /var/log/sysstat/sardd, where dd is the day of the month.
CPU reports
The -P with the processor numbers(s) gives the CPU statistics. -P ALL gives the statistics for all processors.
$ sar -P 0,1 2 2 Linux 3.0.0-14-generic (hostname) Monday 16 July 2012 _i686_ (2 CPU) 12:11:45 IST CPU %user %nice %system %iowait %steal %idle 12:11:47 IST 0 8.50 0.00 1.50 0.00 0.00 90.00 12:11:47 IST 1 3.50 0.00 1.50 0.00 0.00 95.00 12:11:47 IST CPU %user %nice %system %iowait %steal %idle 12:11:49 IST 0 8.00 0.00 1.00 0.00 0.00 91.00 12:11:49 IST 1 6.00 0.00 2.00 0.00 0.00 92.00 Average: CPU %user %nice %system %iowait %steal %idle Average: 0 8.25 0.00 1.25 0.00 0.00 90.50 Average: 1 4.75 0.00 1.75 0.00 0.00 93.50 $ sar -P ALL 2 2 Linux 3.0.0-14-generic (hostname) Monday 16 July 2012 _i686_ (2 CPU) 12:12:06 IST CPU %user %nice %system %iowait %steal %idle 12:12:08 IST all 7.02 0.00 2.01 48.12 0.00 42.86 12:12:08 IST 0 7.50 0.00 1.50 5.00 0.00 86.00 12:12:08 IST 1 6.50 0.00 2.50 91.00 0.00 0.00 12:12:08 IST CPU %user %nice %system %iowait %steal %idle 12:12:10 IST all 6.50 0.00 2.25 45.50 0.00 45.75 12:12:10 IST 0 7.00 0.00 2.50 46.00 0.00 44.50 12:12:10 IST 1 6.03 0.00 2.01 45.23 0.00 46.73 Average: CPU %user %nice %system %iowait %steal %idle Average: all 6.76 0.00 2.13 46.81 0.00 44.31 Average: 0 7.25 0.00 2.00 25.50 0.00 65.25 Average: 1 6.27 0.00 2.26 68.17 0.00 23.31
The CPU column gives the processor id, %user is the percentage time the processor is working in the user mode, %nice is the percent time processor is working in user mode with nice priority, %system is the percent time the processor is working in the kernel mode, %iowait is the percent time the processor is waiting for the I/O to finish, %steal is the percent time processor is waiting while the hypervisor was servicing another virtual processor and %idle is the percent time processor is idle.
Input/Output
The -b option gives the I/O statistics. For example,
$ sar -b 2 4 Linux 3.0.0-14-generic (hostname) Monday 16 July 2012 _i686_ (2 CPU) 09:04:02 IST tps rtps wtps bread/s bwrtn/s 09:04:04 IST 3.50 0.00 3.50 0.00 44.00 09:04:06 IST 0.00 0.00 0.00 0.00 0.00 09:04:08 IST 4.00 0.00 4.00 0.00 52.00 09:04:10 IST 0.00 0.00 0.00 0.00 0.00 Average: 1.88 0.00 1.88 0.00 24.00
The tps column gives the transfers per second issued to the physical devices. The kernel can combine multiple I/O requests into a single transfer. rtps is the total number of read requests per second issued to the physical devices. Similarly, wtps is the total number of write requests per second issued to the physical devices. breads/s is the total number of blocks (1 block = 512 bytes, here) read per second from the devices. And bwrtn/s is the total number of blocks written to the devices per second.
Paging
The -B option gives the paging statistics.
$ sar -B 2 4 Linux 3.0.0-14-generic (hostname) Tuesday 17 July 2012 _i686_ (2 CPU) 12:27:12 IST pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 12:27:14 IST 0.00 0.00 511.00 0.00 1071.50 0.00 0.00 0.00 0.00 12:27:16 IST 0.00 0.00 505.50 0.00 571.00 0.00 0.00 0.00 0.00 12:27:18 IST 0.00 48.00 500.00 0.00 577.50 0.00 0.00 0.00 0.00 12:27:20 IST 0.00 0.00 499.00 0.00 568.50 0.00 0.00 0.00 0.00 Average: 0.00 12.00 503.88 0.00 697.12 0.00 0.00 0.00 0.00
The pgpgin/s statistic gives the number of kilobytes per second paged in from the hard disk. pgpgout/s is number of kilobytes per second paged out to the hard disk. The fault/s statistic is the major and minor page faults occurring per second in the system. The major page faults result in loading of relevant pages from the hard disk. The minor page faults can be resolved without any loading from the disk. The column majflt/s gives the major page faults per second. pgfree/s is the number of pages put on the free list per second by the system. pgscank/s is the total number of pages scanned per second by the kernel swap daemon, kswapd. pgscand/s is the total number of pages scanned directly per second and pgsteal/s is the total number of pages reclaimed from pagecache and swapcache per second. %vmeff is the ratio of pgsteal and pgscan expressed as a percentage and, if non-zero, is a measure of effectiveness of the virtual memory in the system.
Devices
The -d option gives the statistics for each block device. The device names are printed as dev<major-number>-<minor-number>. In the example below, it is dev8-0. Using the -p option (for pretty printing) along with the -d option gives the familiar sda as the device name.
$ sar -d 2 2 Linux 3.0.0-14-generic (hostname) Tuesday 17 July 2012 _i686_ (2 CPU) 08:04:47 IST DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 08:04:49 IST dev8-0 0.50 132.00 0.00 264.00 0.01 16.00 16.00 0.80 08:04:49 IST DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 08:04:51 IST dev8-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util Average: dev8-0 0.25 66.00 0.00 264.00 0.00 16.00 16.00 0.40 $ sar -d -p 2 2 Linux 3.0.0-14-generic (hostname) Tuesday 17 July 2012 _i686_ (2 CPU) 08:05:13 IST DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 08:05:15 IST sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 08:05:15 IST DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 08:05:17 IST sda 7.50 144.00 40.00 24.53 0.04 5.87 3.73 2.80 Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util Average: sda 3.75 72.00 20.00 24.53 0.02 5.87 3.73 1.40
tps is the total number of transfers per second passed on to the device. rd_sec/s is the number of sectors (512 bytes) read per second. wr_sec/s is the number of sectors written per second. avgrq-sz is the average size of requests sent to the device. avgqu-sz is the average length of queue of requests sent to the device. await is the average total time in milliseconds for the requests to the device to get serviced. This includes the waiting time in the queue and the time spent in servicing the requests. svctm is the average time in milliseconds for the I/O requests to get serviced. %util is the percentage of CPU time spent in issuing I/O requests to the device.
Memory
The -R option gives the memory statistics.
$ sar -R 2 4 Linux 3.0.0-14-generic (hostname) Wednesday 18 July 2012 _i686_ (2 CPU) 03:59:52 IST frmpg/s bufpg/s campg/s 03:59:54 IST -2.00 0.00 0.00 03:59:56 IST 0.00 0.00 0.00 03:59:58 IST 0.00 1.00 -1.00 04:00:00 IST 0.00 0.00 1.00 Average: -0.50 0.25 0.00
frmpg/s is the number of memory pages freed by the system per second. A negative value, here, means that the pages have actually been allocated by the system. bufpg/s is the number of additional pages used as buffers by the system per second and campg/s is the number of additional memory pages cached by the system per second.
Swap space
The -S option gives the swap space utilization report.
$ sar -S 2 4 Linux 3.0.0-14-generic (hostname) Wednesday 18 July 2012 _i686_ (2 CPU) 04:37:46 IST kbswpfree kbswpused %swpused kbswpcad %swpcad 04:37:48 IST 2658020 338096 11.28 30140 8.91 04:37:50 IST 2658020 338096 11.28 30140 8.91 04:37:52 IST 2658020 338096 11.28 30140 8.91 04:37:54 IST 2658020 338096 11.28 30140 8.91 Average: 2658020 338096 11.28 30140 8.91
kbswpfree is the kilobytes of free swap space in the system. Similarly, kbswpused is the kilobytes of used swap space in the system. %swpused is the swap space used, expressed as a percentage of the total swap space. kbswpcad is the cached swapped memory in kilobytes. That is, this memory is in the swap area and is also cached by the system. If more memory is required by the system, this can be used straightaway without the need for swapping as it is already in the swap area. %swpcad is the swap memory cached expressed as a percentage of total swap memory used.
Network statistics
It is possible to get elaborate network statistics with sar. The syntax for the network option is,
-n { keyword [,...] | ALL }
The keywords are DEV, EDEV, NFS, NFSD, SOCK, IP, EIP, ICMP, EICMP, TCP, ETCP, UDP, SOCK6, IP6, EIP6, ICMP6 and UDP6. One can get the network statistics based on a list of keywords or, by using the option -n ALL, for all the keywords. For example, the statistics for network interfaces is obtained using the keyword DEV.
$ sar -n DEV 2 2 Linux 3.0.0-17-generic (hostname) Wednesday 18 July 2012 _i686_ (4 CPU) 11:39:43 IST IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 11:39:45 IST lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:39:45 IST eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:39:45 IST wlan0 220.00 191.50 244.69 47.15 0.00 0.00 0.00 11:39:45 IST IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 11:39:47 IST lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:39:47 IST eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:39:47 IST wlan0 249.00 224.00 242.78 47.98 0.00 0.00 0.00 Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: wlan0 234.50 207.75 243.74 47.56 0.00 0.00 0.00
The above table gives a row of output for each interface. The column rxpck/s gives total number of packets received per second. txpck/s gives the total number of packets transmitted per second. rxkB/s is the total number of kilobytes received per second and txkB/s is the total number of kilobytes transmitted per second. rxcmp/s and txcmp/s are the total number of compressed packets received and transmitted per second respectively. rxmcst/s is the total number of multicast packets received per second.
Next, let us look at the IP statistics. For IPV4 statistics to be collected, sadc should have been started with the -S SNMP
option.
$ sar -n IP 2 2 Linux 2.6.38-8-virtual (hostname) 07/18/2012 _i686_ (1 CPU) 06:11:14 AM irec/s fwddgm/s idel/s orq/s asmrq/s asmok/s fragok/s fragcrt/s 06:11:16 AM 2.38 0.00 2.38 1.36 0.00 0.00 0.00 0.00 06:11:18 AM 2.78 0.00 2.78 2.02 0.00 0.00 0.00 0.00 Average: 2.61 0.00 2.61 1.74 0.00 0.00 0.00 0.00
irec/s is the total number of input datagrams received on the interfaces per second (ipInReceives). This includes datagrams received due to errors. fwddgm/s is the number of datagrams received per second for which this host was not the final destination and, therefore, an attempt was made to find a route and forward these datagrams to the final destination (ipForwDatagrams). idel/s is the number of input datagrams delivered per second to upstream protocols (ipInDelivers). orq/s is the number of datagrams received from upstream protocols per second for transmission (ipOutRequests). asmrq/s is the number of IP fragments received per second which need to be re-assembled (ipReasmReqds). asmok/s is the number of IP datagrams re-assembled successfully per second (ipReasmOKs). fragok/s is number of IP datagrams that have been fragmented successfully per second here (ipFragOKs). fragcrt/s is the number of IP datagram fragments generated per second as a consequence of fragmentation (ipFragCreates).
As another example, we look at the TCP statistics. For sadc to collect TCPv4 statistics, sadc should have been started with the option -S SNMP
.
$ sar -n TCP 2 2 Linux 2.6.38-8-virtual (hostname) 07/18/2012 _i686_ (1 CPU) 06:09:05 AM active/s passive/s iseg/s oseg/s 06:09:07 AM 0.00 0.00 6.28 8.47 06:09:09 AM 0.00 1.25 19.05 20.30 Average: 0.00 0.65 12.94 14.64
active/s is the number of times TCP connections have made a direct transition from the CLOSED state to the SYN-SENT state per second (tcpActiveOpens). passive/s is the number of times TCP connections have made a direct transition from the LISTEN state to the SYN-RECEIVED state per second (tcpPassiveOpens). iseg/s is the total number of segments received per second including those received due to error (tcpInSegs). oseg/s is the total number of segments sent per second excluding the ones that contain octets for retransmission only (tcpOutSegs).
The last example for network statistics uses the keyword, SOCK. This gives statistics for sockets being used in the system.
$ sar -n SOCK 2 2 Linux 3.0.0-14-generic (hostname) Wednesday 18 July 2012 _i686_ (2 CPU) 08:50:04 IST totsck tcpsck udpsck rawsck ip-frag tcp-tw 08:50:06 IST 717 20 8 0 0 0 08:50:08 IST 717 20 8 0 0 0 Average: 717 20 8 0 0 0
The first column gives the total sockets in the system. The next two columns give the TCP and UDP sockets which are in use. The last three columns give raw sockets in use, IP fragments in queue and TCP sockets in TIME_WAIT state respectively. The last five columns do not add up to total sockets. This should not be a worry since there are many more types of sockets in the system, which are not reported by sar.
Besides above examples, EDEV keyword gives statistics on errors in the network devices, EIP gives IPv4 network error reports, EICMP gives ICMPv4 error reports and ETCP gives TCPv4 error reports. Similar to IPv4, there are IPv6 keywords.
Run queue length and load average
With the -q option, the run queue length and load averages are obtained.
$ sar -q 2 2 Linux 3.0.0-14-generic (hostname) Thursday 19 July 2012 _i686_ (2 CPU) 10:15:05 IST runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 10:15:07 IST 1 446 0.30 0.15 0.09 1 10:15:09 IST 2 445 0.28 0.15 0.09 2 Average: 2 446 0.29 0.15 0.09 2
runq-sz is the number of tasks (processes and threads) on the run queue, waiting for the processor. plist-sz is the total number of currently live tasks. ldavg-1, ldavg-5 and ldavg-15 are the load averages for the last one, five and fifteen minutes respectively. blocked is the number of tasks blocked, waiting for the I/O to be completed.
Interrupts
The syntax for the option to get interrupt statistics is,
-I {int, [,...] | SUM | ALL | XALL }
For data collection for interrupt statistics, sadc should have been started with the option, -S INT
. A list of interrupt numbers needs to be passed with the -I option for generating the interrupt statistics report. The keyword SUM gives the total number of interrupts received per second. ALL gives a report on first sixteen interrupts whereas XALL gives the report on all interrupts received per second.
$ sar -I SUM 2 2 Linux 3.0.0-14-generic (hostname) Thursday 19 July 2012 _i686_ (2 CPU) 12:30:57 IST INTR intr/s 12:30:59 IST sum 1016.50 12:31:01 IST sum 964.50 Average: sum 990.50
Power management
The -m option gives the power management statistics. For data collection for power management reports, it is required that sadc should be started with the option -S POWER
. The syntax for the option parameter is,
-m { keyword [, ...] | ALL }
The keywords are CPU, FAN, FREQ, IN, TEMP and USB. With the CPU keyword, instantaneous CPU clock frequency in MHz is reported. The FAN keyword reports the fan speed in rpm, the difference between the current fan speed and its low limit in rpm and the sensor device name. The FREQ keyword reports the weighted average CPU clock frequency in MHz. The IN keyword reports details about voltage inputs. It gives input voltage in volts, relative input voltage value between high and low limits, and the sensor device name. The TEMP keyword gives the report on devices temperature. It gives the device temperature in degrees Celcius, relative device temperature and the sensor device name. The USB keyword gives a report on all the USB devices plugged in to the system. It gives the root hub number of the USB device, the vendor id number, product id number, maximum power consumption of the device, the manufacturer's name and the product name.
$ sar -m USB 1 1 Linux 3.0.0-17-generic (hostname) Thursday 19 July 2012 _i686_ (4 CPU) 01:25:41 IST BUS idvendor idprod maxpower manufact product 01:25:42 IST 1 8087 20 0 01:25:42 IST 2 8087 20 0 01:25:42 IST 1 c45 6409 1000 USB 2.0 Camera_FLV14 USB 2.0 Camera 01:25:42 IST 1 489 e00f 0 Broadcom Corp Broadcom Bluetooth Device 01:25:42 IST 2 3f0 4817 196 Hewlett-Packard HP LaserJet P1007 Summary BUS idvendor idprod maxpower manufact product Summary 1 8087 20 0 Summary 2 8087 20 0 Summary 1 c45 6409 1000 USB 2.0 Camera_FLV14 USB 2.0 Camera Summary 1 489 e00f 0 Broadcom Corp Broadcom Bluetooth Device Summary 2 3f0 4817 196 Hewlett-Packard HP LaserJet P1007
