Monitoring Linux System

Linux Essentials for DevOps Automation

Linux monitoring involves keeping track of system resources, performance metrics, and application health to ensure optimal operation and troubleshoot issues.

Here’s a comprehensive guide to Linux monitoring:

1. System Monitoring Tools:

top: Provides a dynamic real-time view of system processes, CPU usage, memory usage, and more.

top - 07:04:42 up 15:28, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 273 total, 1 running, 272 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.2 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 1639.8 total, 365.5 free, 662.1 used, 712.7 buff/cache
MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 977.7 avail Mem 

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
822 root 20 0 459244 8424 6760 S 0.3 0.5 2:26.35 vmtoolsd 
35368 root 20 0 226188 3840 3072 R 0.3 0.2 0:00.04 top 
1 root 20 0 108384 15312 9476 S 0.0 0.9 0:03.47 systemd 
2 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kthreadd 
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp 
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp 
5 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 slub_flushwq 
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns 
10 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq 
12 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tasks_kthre 
13 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tasks_rude_ 
14 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tasks_trace

vmstat: Reports information about system memory, processes, and CPU usage.

Usage: vmstat [interval] [count]

[root@uadev ~]# vmstat 
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 375012 5360 724428 0 0 6 2 20 38 0 0 100 0 0
[root@uadev ~]#

2. Disk Monitoring:

df: Displays disk space usage for mounted filesystems.

Usage: df -h

[devnixops@uadev tmp]$ df -h 
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 820M 100K 820M 1% /dev/shm
tmpfs 328M 6.5M 322M 2% /run
efivarfs 256K 31K 226K 13% /sys/firmware/efi/efivars
/dev/mapper/cs-root 17G 4.3G 13G 27% /
/dev/nvme0n1p2 960M 271M 690M 29% /boot
/dev/nvme0n1p1 599M 7.0M 592M 2% /boot/efi
tmpfs 164M 52K 164M 1% /run/user/42
tmpfs 164M 44K 164M 1% /run/user/1000
[devnixops@uadev tmp]$

du: Estimates file and directory space usage.

Usage: du -h [directory]

[root@uadev var]# du -h log
0 log/private
0 log/samba/old
0 log/samba
192K log/audit
4.0K log/sssd
0 log/speech-dispatcher
12K log/cups
0 log/gdm
0 log/qemu-ga
0 log/tuned
0 log/chrony
6.9M log/anaconda
7.8M log
[root@uadev var]#

3. Network Monitoring:

netstat: Displays network connections, routing tables, interface statistics, and more.

Usage: netstat -tuln

[root@uadev var]# netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 
tcp6 0 0 ::1:631 :::* LISTEN 
tcp6 0 0 :::22 :::* LISTEN 
udp 0 0 0.0.0.0:56178 0.0.0.0:* 
udp 0 0 0.0.0.0:5353 0.0.0.0:* 
udp 0 0 127.0.0.1:323 0.0.0.0:* 
udp6 0 0 :::5353 :::* 
udp6 0 0 ::1:323 :::* 
udp6 0 0 :::34280 :::* 
[root@uadev var]#

4. Process Monitoring:

ps: Displays information about active processes.

Usage: ps aux

[root@uadev var]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.9 108384 15312 ? Ss Jun05 0:03 /usr/lib/systemd/systemd rhgb --switched-root --system --
root 2 0.0 0.0 0 0 ? S Jun05 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? I< Jun05 0:00 [rcu_gp]
root 4 0.0 0.0 0 0 ? I< Jun05 0:00 [rcu_par_gp]
root 5 0.0 0.0 0 0 ? I< Jun05 0:00 [slub_flushwq]
root 6 0.0 0.0 0 0 ? I< Jun05 0:00 [netns]
root 10 0.0 0.0 0 0 ? I< Jun05 0:00 [mm_percpu_wq]
root 12 0.0 0.0 0 0 ? I Jun05 0:00 [rcu_tasks_kthre]

pgrep: Searches for processes based on name and other attributes.

Usage: pgrep process_name

[root@uadev var]# pgrep auditd
28
781
[root@uadev var]#

5. Log Monitoring:

tail: Displays the last few lines of a file (e.g., log files).

Usage: tail -f /var/log/messages

[root@uadev ~]# tail -f /var/log/messages
Jun 6 06:42:13 uadev systemd[1]: Started Hostname Service.
Jun 6 06:42:43 uadev systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Jun 6 06:50:51 uadev systemd[1]: Starting PackageKit Daemon...
Jun 6 06:50:52 uadev systemd[1]: Started PackageKit Daemon.
Jun 6 06:50:52 uadev dbus-broker[805]: A security policy denied :1.25 to send method call /org/freedesktop/PackageKit:org.freedesktop.DBus.Properties.GetAll to :1.168.
Jun 6 06:51:48 uadev NetworkManager[926]: <info> [1717636908.3057] dhcp4 (ens160): state changed new lease, address=192.168.22.131
Jun 6 06:51:48 uadev systemd[1]: Starting Network Manager Script Dispatcher Service...
Jun 6 06:51:48 uadev systemd[1]: Started Network Manager Script Dispatcher Service.
Jun 6 06:51:58 uadev systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Jun 6 06:55:57 uadev systemd[1]: packagekit.service: Deactivated successfully.
^C
[root@uadev ~]#

grep: Searches for patterns in files, often used with logs.

Usage: grep "pattern" /var/log/messages


[root@uadev ~]# grep error /var/log/messages
Jun 5 14:54:44 uadev cupsd[981]: REQUEST localhost - - "POST / HTTP/1.1" 200 182 Renew-Subscription client-error-not-found
Jun 5 16:17:26 uadev cupsd[981]: REQUEST localhost - - "POST / HTTP/1.1" 200 182 Renew-Subscription client-error-not-found
Jun 5 17:15:46 uadev cupsd[981]: REQUEST localhost - - "POST / HTTP/1.1" 200 182 Renew-Subscription client-error-not-found
Jun 5 21:48:00 uadev cupsd[981]: REQUEST localhost - - "POST / HTTP/1.1" 200 182 Renew-Subscription client-error-not-found
Jun 5 22:46:20 uadev cupsd[981]: REQUEST localhost - - "POST / HTTP/1.1" 200 182 Renew-Subscription client-error-not-found
[root@uadev ~]#

6. Performance Monitoring:

sar: Collects, reports, and saves system activity information.

Usage: sar

[root@uadev ~]# sar -u 1 5 
Linux 5.14.0-447.el9.aarch64 (uadev) 06/06/2024 _aarch64_ (2 CPU)

07:08:52 AM CPU %user %nice %system %iowait %steal %idle
07:08:53 AM all 0.00 0.00 0.00 0.00 0.00 100.00
07:08:54 AM all 0.00 0.00 0.50 0.00 0.00 99.50
07:08:55 AM all 0.00 0.00 0.00 0.00 0.00 100.00
07:08:56 AM all 0.00 0.00 0.00 0.00 0.00 100.00
07:08:57 AM all 0.50 0.00 0.00 0.00 0.00 99.50
Average: all 0.10 0.00 0.10 0.00 0.00 99.80
[root@uadev ~]#

iostat: Reports CPU utilization and input/output statistics for devices, partitions, and more.

Usage: iostat

[root@uadev ~]# iostat 1 5 
Linux 5.14.0-447.el9.aarch64 (uadev) 06/06/2024 _aarch64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.01 0.11 0.00 0.00 99.77

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
dm-0 0.61 10.57 4.44 0.00 591627 248517 0
dm-1 0.00 0.04 0.00 0.00 2216 0 0
nvme0n1 0.59 11.64 4.48 0.00 651733 250615 0

7. Application Monitoring:

Systemd: Manages system services and can monitor their status.

Usage: systemctl status service_name

[root@uadev ~]# systemctl status sshd
● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; preset: enabled)
Active: active (running) since Mon 2024-05-27 22:57:52 IST; 1 week 2 days ago
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 982 (sshd)
Tasks: 1 (limit: 10116)
Memory: 5.0M
CPU: 78ms
CGroup: /system.slice/sshd.service
└─982 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

May 27 22:57:52 uadev systemd[1]: Starting OpenSSH server daemon...
May 27 22:57:52 uadev sshd[982]: Server listening on 0.0.0.0 port 22.
May 27 22:57:52 uadev sshd[982]: Server listening on :: port 22.

Prometheus: Open-source monitoring and alerting toolkit.

Usage: Configured with exporters to collect metrics from various services.

Linux monitoring is essential for maintaining system health, identifying performance bottlenecks, and troubleshooting issues. By leveraging the wide range of monitoring tools and techniques available, administrators can ensure the reliability, stability, and performance of their Linux-based systems and applications. Regular monitoring and analysis of system metrics enable proactive management and optimization of Linux environments.