Monitoring services using journalctl in RHEL7

Monitoring services using journalctl in RHEL7

Monitoring services using journalctl in RHEL7

Systemd's journal has the added advantage that its controls allow you to easily narrow down on messages generated by specific services.

1. First, display all the messages generated by your system.

This will show all the messages generated on the system; run the following commands:

[root@nsk ~]# journalctl

-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:32:09 IST. --

Dec 26 07:31:23 nsk systemd-journal[89]: Runtime journal is using 8.0M (max allowed 91.9M, trying to leave 137.9M free of 911.6M available → current limit 91.

Dec 26 07:31:23 nsk kernel: Initializing cgroup subsys cpuset

Dec 26 07:31:23 nsk kernel: Initializing cgroup subsys cpu

Dec 26 07:31:23 nsk kernel: Initializing cgroup subsys cpuacct

Dec 26 07:31:23 nsk kernel: Linux version 3.10.0-693.5.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #

Dec 26 07:31:23 nsk kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.5.2.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root

Dec 26 07:31:23 nsk kernel: e820: BIOS-provided physical RAM map:

Dec 26 07:31:23 nsk kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable

Dec 26 07:31:23 nsk kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved…...

…………..

~]#

2. Now, display all system-related messages.

This command shows all the messages related to the system and not its users:

[root@nsk ~]# journalctl –-system

3. Display all the current user messages.

This command shows all messages related to the user that you are logged on with:

[root@nsk ~]# journalctl –-user

4. Next, display all messages generated by a particular service using the following command line:

journalctl --unit=<service>

[root@nsk ~]# journalctl --unit=sshd

-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --

Dec 26 07:31:29 nsk systemd[1]: Starting OpenSSH server daemon...

Dec 26 07:31:29 nsk sshd[944]: Server listening on 0.0.0.0 port 22.

Dec 26 07:31:29 nsk sshd[944]: Server listening on :: port 22.

Dec 26 07:31:29 nsk systemd[1]: Started OpenSSH server daemon.

Dec 26 07:33:37 nsk sshd[1238]: Accepted password for root from 10.0.2.2 port 60698 ssh2

Dec 26 07:34:11 nsk sshd[1261]: Accepted password for root from 10.0.2.2 port 60702 ssh2

Dec 26 08:30:19 nsk systemd[1]: Stopping OpenSSH server daemon...

……..

6. Now, display messages by priority.

Priorities can be specified by a keyword or number, such as debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0). When specifying a priority, this includes all the lower priorities as well. For example, err implies that crit, alert, and emerg are also shown. Take a look at the following command line:

journalctl -p <priority>

[root@nsk ~]# journalctl -p err

-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --

Dec 26 08:26:15 nsk rsyslogd[613]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]

Dec 26 08:30:21 nsk lvmetad[483]: Failed to accept connection errno 11.

7. Next, display messages by time.

You can show all messages from the current boot through the following commands:

[root@nsk ~]# journalctl -b

-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --

Dec 26 08:30:45 nsk systemd-journal[86]: Runtime journal is using 8.0M (max allowed 91.9M, trying to leave 137.9M free of 911.6M available → current limit 91.

Dec 26 08:30:45 nsk kernel: Initializing cgroup subsys cpuset

Dec 26 08:30:45 nsk kernel: Initializing cgroup subsys cpu

Dec 26 08:30:45 nsk kernel: Initializing cgroup subsys cpuacct

Dec 26 08:30:45 nsk kernel: Linux version 3.10.0-693.5.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #

Dec 26 08:30:45 nsk kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.5.2.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root

Dec 26 08:30:45 nsk kernel: e820: BIOS-provided physical RAM map:

You can even show all the messages within a specific time range by running the following:

[root@nsk ~]# journalctl --since="2017-12-26 08:30:00" --until="2017-12-26 09:00:00"

-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --

Dec 26 08:30:19 nsk polkitd[619]: Registered Authentication Agent for unix-process:1587:353648 (system bus name :1.40 [/usr/bin/pkttyagent --notify-fd 5 --fal

Dec 26 08:30:19 nsk ntpd[1551]: ntpd exiting on signal 15

Dec 26 08:30:19 nsk systemd[1]: Stopped target Network is Online.

Dec 26 08:30:19 nsk sshd[944]: Received signal 15; terminating.

Dec 26 08:30:19 nsk systemd[1]: Stopping Network is Online.

Dec 26 08:30:19 nsk crond[628]: (CRON) INFO (Shutting down)

…….

For instance, if you want to show all the error messages between 8:30 and 9:00 on 2017-12-26, your command would be the following:

[root@nsk ~]# journalctl -p err --since="2017-12-26 08:30:00" --until="2017-12-26 09:00:00"

-- Logs begin at Tue 2017-12-26 07:31:23 IST, end at Tue 2017-12-26 08:33:14 IST. --

Dec 26 08:30:21 nsk lvmetad[483]: Failed to accept connection errno 11.

[root@nsk ~]#

The journalctl binary is an executable one, so it is impossible to use the traditional "following" techniques such as tail –f or using less and pressing CTRL + F. Simply add -f or --follow as an argument to the journalctl command.

[root@nsk ~]# journalctl -f

-- Logs begin at Tue 2017-12-26 07:31:23 IST. --

Dec 26 08:30:53 nsk systemd[1]: Started Crash recovery kernel arming.

Dec 26 08:30:53 nsk systemd[1]: Startup finished in 392ms (kernel) + 1.702s (initrd) + 6.681s (userspace) = 8.777s.

Dec 26 08:32:08 nsk sshd[1259]: Accepted password for root from 10.0.2.2 port 63824 ssh2

Dec 26 08:32:09 nsk systemd[1]: Created slice User Slice of root.

Dec 26 08:32:09 nsk systemd[1]: Starting User Slice of root.

Although most environments are used to create syslog messages to troubleshoot, the journal does provide the added value of being able to create simple filters that allow you to monitor their messages live.