Understanding and Configuring System Logging

{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}

Understanding System Logging

Services on a Linux server write information to log files in different destinations using different approaches, and there are multiple ways to find relevant information in those log files.

Services can write log information via:

• Direct Write - information is written directly to a log file by a service.
• rsyslogd - a service passes log information to the rsyslogd service that in turn takes care of managing centralized log files.
• journald - this service is tightly integrated with Systemd.

journald receives all messages generated by Systemd Units. It also collects messages from the kernel, the boot procedure and services, then writes these message to an event journal which is stored in binary format. This journal can be queried with the journalctl command.

journald is not persistent between reboots, so messages are also forwarded to rsyslogd which in turn writes the log information to different files in the /var/log directory and thus making them persistent.

rsyslogd adds some services to journald, it allows you to configure remote logging and log servers, and makes logging persistent.

We also have the auditd service which offers an in-depth trace of what specific services, processes and users have been doing.

We the above in mind, we have three approaches to get more information on what has been happening on a machine:

• Monitor the files in /var/log written by rsyslogd or by services directly.
• Use the journalctl command.
• Use the systemctl status command.

Persistent Log Files

As mentioned earlier, persistent log files are located in the /var/log directory. Most of the files are managed by rsyslogd but some of them are created directly by services. The most convenient way of reading these files would be by using a pager like less.

The below table provides an overview of some of the standard files that are created in this directory and what content you may expect.

{{

}} Log File | Explanation ---------|------------ /var/log/messages | A generic log file where most messages are written to. /var/log/dmesg | Contains kernel log messages. /var/log/secure | Contains authentication-related messages. /var/log/boot.log | Messages related to system startup /var/log/audit/audit.log | Contains audit messages. SELinux writes to this file. /var/log/maillog | Mail-related messages /var/log/samba | The Samba service writes directly to this file, it's not managed through rsyslogd. /var/log/sssd | Messages written by the sssd service, which plays an important role in the authentication process. /var/log/cups | Messages generated the the CUPS print service. /var/log/httpd | Directory that contains log files written directly by the Apache web server. {{}}

It might be useful to see what is happening in real time. The tail -f <logfile> command would allow you to live monitor a specific log file, you'll see lines being added to the output of the command as system events happen.

Using logger

While services write information to log files by themselves or through rsyslogd, users can write messages to rsyslogd from the command line or a script using the logger command. Simply type logger followed by the message you want to write to the logs.

You can also specify the priority and facility to log to. For example, logger -p kern.err hello would write hello to the kernel facility using the error priority.

Configuring rsyslogd

The rsyslogd service makes sure that information that needs to be logged is written to the location where you would want to find it. This is configured inside the /etc/rsyslog.conf file that acts as the central location from where rsyslogd is configured. From this file the /etc/rsyslog.d/ directory is included, RPM packages can put specific configuration files inside this directory.

If you need to pass specific options to the rsyslogd service on startup you need to do that via the /etc/sysconfig/ryslog file. This file contains one line by default where you can specify startup parameters: SYSLOGD_OPTIONS="" This variable is included in the Systemd configuration file that starts rsyslogd.

Understanding rsyslogd Configuration Files

The rsyslogd.conf file contains different sections to specify what should be logged and where:

• ### MODULES ###: rsyslogd is modular and you can expand features using modules.

• ### GLOBAL DIRECTIVES ###: Specified global parameters like the default timestamp format or the location where files are being written to.

• ### RULES ###: This is the most important part, it contains rules that specify which information should be logged to which destination.

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Facilities, Priorities and Destinations

The Rules section we discussed above uses facilities, priorities and destinations to specify what should be logged and where.

• A facility specifies a category of information. There is a fixed list of facilities that cannot be extended to ensure backwards compatibility. As a result some facilities exist that serve no purpose anymore and some services that became relevant at a later stage do not have their own facility. To circumvent that you can configure the service to use the generic daemon and local0 to local7 facilities.

{{

}} Facility | Used by -----|---- auth/authpriv | Messages related to authentication cron | Messages generated by the crond service daemon | Generic facility to be used for nonspecified daemons kern | Kernel messages lpr | Messages generated through the legacy lpd print system mail | Email related messages mark | Special facility that can be used to periodically write a marker news | Messages generated by the NNTP news system security | Deprecated, same as auth/authpriv syslog | Messages generated by the syslog system user | Messages generated in user space uucp | Messages generated by the legacy UUCP system local0-7 | Messages generated by services configured to use one of the local0 to local7 facilities {{}}

• Priorities define the severity of the message that needs to be logged. When specifying a priority, all messages with that specific priority and higher will be logged.
  If you need to configure logging where messages with different priorities are sent to different files you can specify the priority with a = sign: cron.=debug -/var/log/cron.debug This will write all cron message with only the debug priority to the /var/log/cron.debug log.

{{

}} Priority | Description -----|---- emerg/panic | Message generated when the availability of the service is discontinued alert | Used when the availability of the service is about to be discontinued crit | A critical error has occurred error/err | A noncritical error has occurred warning/warn | Something is suboptimal, but no real error yet notice | Messages about items that might become an issue later info | Informational messages about normal service operation debug | Debug messages that will give as much information as possible about service operation {{}}

• Destinations defines where the message should be written to. Typically these are files, but they can be devices or ryslog modules as well. If the destination file name starts with a hypen, e.g. -/var/log/maillog, the message will not be commited immediately to the file. Instead, it will be buffered to make writes more efficient.

See man 5 rsyslog.conf

Rotating Log Files

When a certain treshold has been reached the old log file is closed and a new log file is opened by the logrotate utility which is started periodically by the crond service.

When a log file is rotated, the old log file is typically copied to a new file that has the rotation date in it. By default, four old log files are kept on the system, files older than that period are removed from the system automatically.

You should back up log files or configure a centralized log server where logrotate keeps rotated messages for a significanlty longer period.

The default settings for log rotation are kep in the /etc/logrotate.conf file, but you can create configuration files inside the /etc/logrotate.d/ directory to be applied to specific log files.

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

Working with journald

The systemd-journald service stores log messages in a binary file, the journal, /run/log/journal. Since this file is in the /run directory, it is not persistent.

We can examine the journal using the journalctl command, the output is shown in a less pager and by default you'll see the beginning of the journal. You can live monitor the journal using the journcalctl -f command.

journalctl has many filtering options, you can find some of the most useful in the table below. You can also use tab completion to see specific options.

{{

}} Command | Description -----|---- journalctl --no-pager | Shows the content without using a pager journalctl -f | Live monitor the journal as events happen journalctl _UID=1000 | Show messages that have been logged for a specific user journalctl -p err | Show messages with the error priority journalctl --since yesterday --until today | Shows messages in a specific time frame. Accepts keywords like yesterday, today, tomorrow or you can specify a date/time in the YYYY-MM-DD hh:mm:ss format. journalctl _SYSTEMD_UNIT=sshd.service | Shows information on the sshd Systemd Unit. journalctl --dmesg | Shows kernel related messages journalctl -o verbose | Shows detailed information on logged items {{}}
The above options can all be combined, e.g. journalctl --since yesterday -p err _UID=1000 or journalctl _SYSTEMD_UNIT=sshd.service -o verbose.

Persistently Storing the Systemd Journal

Storing the journal in a persistent way so that it survives reboots requires setting the Storage= parameter inside /etc/systemd/journal.conf:

• Storage=auto The journal will be written to disk if the /var/log/journal directory exists.
• Storage=volatile The journal will only be stored in /run/log/journal in a non persistent way.
• Storage=persistent The journal will be written to disk in the /var/log/journal directory, if this directory does not exist it will be created.
• Storage=none No data will be stored, but forwarding to other targets like the kernel log buffer or syslog will still work.

If you manually created the /var/log/journal directory you need to set ownership and permissions, then either reboot or kill the systemd-journald service. Restarting the systemd-journald service is not enough:

# chown root:systemd-journal /var/log/journal
# chmod 2755 /var/log/journal
# killall -USR1 systemd-journald

The journal has built-in log rotation, so even if the journal is stored persistently the data is not kept forever. The journal is limited to a maximum size of 10% of the file system it's on and it will stop growing when less than 15% of file system is free. When this happens the oldest messages from the journal are dropped to make room for new messages. This can be changed in the /etc/systemd/journal.conf file.