An Introduction to Systemd and Systemd Unit files

{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}

Systemd is considered to be the future standard init system by all mainstream Linux distributions. The Systemd System and Service Manager uses "units" as an abstraction for parts of the system to be managed: typically services, sockets, mounts and targets. It provides a uniform interface for managing units.

The systemctl -t help command displays a list of available units that can be managed with Systemd.

[student@server1 ~]$ systemctl -t help
Available unit types:
service
mount
swap
socket
target
device
automount
timer
path
slice
scope

Each unit type can be recognized by the file extension. A service unit will end on .service while a target unit will end on .target.

Unit files are located in three different locations:

/usr/lib/systemd/system contains default unit files that have been installed by RPM packages. These should not be edited since package updates can overwrite your changes.
/etc/systemd/system contains custom unit files written by a system administrator or generated by the systemctl edit command.
/run/systemd/system contains unit files that have been generated automatically by the system during runtime.

The above locations have a specific priority if a unit file happens to exist in more than one location. Units in the /run directory have the highest priority, followed by custom unit files in the /etc/systemd/system directory and lastly the files inside /usr/lib/systemd/system.

Systemd Service Units

The most important unit type is the service unit. You can start any type of process by using a service unit, including daemon processes and regular commands.

Let's take a look at the service unit file for the Very Secure Ftp Daemon:

[root@server1 ~]# systemctl cat vsftpd
# /usr/lib/systemd/system/vsftpd.service
[Unit]
Description=Vsftpd ftp daemon
After=network.target

[Service]
Type=forking
ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

[Install]
WantedBy=multi-user.target

We can identify three sections in this service unit file:

[Unit] This sections describes the unit and dependencies (more on that later). It contains the important After statement and optional Before statement. These statements define dependencies between different units and how they relate from the perspective of this unit. In the above example, the After statement indicates that this unit should be started after the network.target unit.
[Service] describes how to start and stop the unit. Usually you will see the ExecStart statement to indicate how to start the unit and the ExecStop statement for how to stop the unit. The Type statement is used to specify how the process should start. The forking type is commonly used by daemon processes.
[Install] indicates in which target unit this service has to start. We'll cover more on target units later.

For more detailed information on the above, see man 5 systemd.service.

Systemd Mount Units

A mount unit specifies how a file system can be mounted on a specific directory. Configuring mount points through /etc/fstab is the preferred approach, any mount points configured in the /etc/fstab file will be converted to a Systemd mount unit at boot time.

[root@server1 ~]# systemctl cat tmp.mount
[Unit]
Description=Temporary Directory (/tmp)
Documentation=https://systemd.io/TEMPORARY_DIRECTORIES
Documentation=man:file-hierarchy(7)
Documentation=https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
ConditionPathIsSymbolicLink=!/tmp
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=tmpfs
Where=/tmp
Type=tmpfs
Options=mode=1777,strictatime,nosuid,nodev</code></pre>

We typically see the following options inside the [Mount] section:

What takes an absolute path of a file or device node. This is a mandatory option.
Where defines the absolute path for the mount point, which cannot be a symbolic link. If it does not exist at the time of mounting, it is created as a directory. This is a mandatory option as well and the string must reflect the unit filename.
Type defines the file system type. This is optional.

See man 5 systemd.mount for more details.

Systemd Socket Units

A socket creates a method for applications to communicate either by means of a file or a TCP/UDP port on which Systemd will be listening for incoming connections.

This means a specific service doesn't have to be running continuously, if Systemd detects an incoming connection on the socket it can start the associated service (on-demand starting). With that in mind, each socket unit must have a corresponding service unit file.

[root@server1 ~]# systemctl cat cockpit.socket
# /usr/lib/systemd/system/cockpit.socket
[Unit]
Description=Cockpit Web Service Socket
Documentation=man:cockpit-ws(8)
Wants=cockpit-motd.service

[Socket]
ListenStream=9090
ExecStartPost=-/usr/share/cockpit/motd/update-motd '' localhost
ExecStartPost=-/bin/ln -snf active.motd /run/cockpit/motd
ExecStopPost=-/bin/ln -snf /usr/share/cockpit/motd/inactive.motd /run/cockpit/motd

[Install]
WantedBy=sockets.target

The important option in the above example is ListenStream. This options specifies the address and/or TCP port number to listen on. It can be written as e.g. 192.168.0.1:9090 or 9090 or /path/to/file.socket or [ipv6]:portnumber

For UDP ports you would use ListenDatagram instead.

See man 5 systemd.socket for more information.

Systemd Target Units

A target unit makes it possible to load unit files in a specific order to define the state the machine should be started in, very similar to the runlevels used in other init systems. A target unit is basically a group of units, they don't have additional functionality on top of the units they group.

[root@server1 ~]# systemctl cat multi-user.target
# /usr/lib/systemd/system/multi-user.target

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes

The target unit has definitions on what it requires and what other targets it cannot coexist with (Conflicts). It defines load ordering by using the After option.

When using the systemctl enable command on a unit to automatically start it at boot time, the [Install] section of that unit determines to what target unit it should be added. Behind the scenes, the systemctl enable command creates a symbolic link in the target directory (defined by [Install]) inside /etc/systemd/system.

For example, systemctl enable vsftpd adds a symbolic link in /etc/systemd/system/multi-user.target/wants/vsftpd.service which points to /usr/lib/systemd/system/vsftpd.service ensuring the vsftpd.service unit will be started automatically with the Multi-User System target. This symbolic link is called a want, it defines what the target wants to start when it is processed.

[root@server1 ~]# systemctl enable vsftpd
Created symlink /etc/systemd/system/multi-user.target.wants/vsftpd.service → /usr/lib/systemd/system/vsftpd.service.

Using Systemd to Manage Units

You should've noticed we can use the systemctl command to manage Systemd units. systemctl start/stop will start/stop a unit, while systemctl enable/disable will either start the unit at boot or prevent the unit from starting at boot.

The systemctl status command shows us runtime status information on the unit as well as the most recent journal log data:

[root@server1 ~]# systemctl status vsftpd
● vsftpd.service - Vsftpd ftp daemon
     Loaded: loaded (/usr/lib/systemd/system/vsftpd.service; disabled; vendor preset: disabled)
     Active: active (running) since Thu 2020-06-25 14:27:13 +04; 1s ago
    Process: 66451 ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf (code=exited, status=0/SUCCESS)
   Main PID: 66452 (vsftpd)
      Tasks: 1 (limit: 28430)
     Memory: 636.0K
        CPU: 4ms
     CGroup: /system.slice/vsftpd.service
             └─66452 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

Jun 25 14:27:13 DELLG5 systemd&#91;1]: Starting Vsftpd ftp daemon...
Jun 25 14:27:13 DELLG5 systemd&#91;1]: Started Vsftpd ftp daemon.

The second line shows us the unit has been loaded into memory, but is not enabled at boot. The status of Loaded can also be error, not-found, bad-setting, masked or static.

The Active line shows the current state:

running: The unit is running with active processes
exited: A one time run was successfully completed.
waiting: The unit is running and waiting for events.
inactive (dead): The unit is not running

In addition to the systemctl status command for a specific unit, the following commands can help you get a bigger picture of unit statuses:

{{

}} Command | Description -----|---- systemctl –type=service | Show service units systemctl list-units –type-service | idem systemctl list-units –type=service –all | Show active and inactive service units systemctl –failed –type=service | Show failed services {{

}}

Managing Systemd Dependencies

Unit files can have dependencies, a unit file may Want, Require or Requisite one or more other units After or Before it can run. These keywords can be used in the [Unit] section of a unit file.

Requires: If this unit loads, then units specified here will also load. Deactivating either unit will result in both units being deactivated.
Requisite: If the unit listed here is not already loaded, then the unit will fail.
Wants: The unit will try to load units specified here, but it will not fail if any of the specified units do fail.
Before: The unit will start before the unit listed here.
After: The unit will start after the unit listed here.

You can request a list of unit dependencies using the systemctl list-dependencies command. Adding a unit to the command will show dependencies for that specific unit, and adding the - -reverse flag will show what units are dependent on that specific unit.

[root@server1 ~]# systemctl list-dependencies vsftpd
vsftpd.service
● ├─system.slice
● └─sysinit.target
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ├─dmraid-activation.service
....

[oot@server1 ~]# systemctl list-dependencies multi-user.target
multi-user.target
● ├─abrt-journal-core.service
● ├─abrt-oops.service
....

[root@server1 ~]# systemctl list-dependencies --reverse multi-user.target
multi-user.target
● └─graphical.target

Managing Unit Options

There's a huge amount of options available for each unit file, to see what options (and their default values) are available for a specific unit use the systemctl show command.

Changes to unit files need to be written to the /etc/systemd/system directory which is where custom unit files are located. The recommended way is to do so by using the systemctl edit command on a unit, this will create an override file for the unit and store that in the correct location. All settings in this file overwrite existing settings in /usr/lib/systemd/system.

By default, systemctl edit will use the nano editor, but you can change the default file editor by including the following in ~/.bash_profile or /etc/bash_profile to change this system wide:
export SYSTEMD_EDITOR="/bin/vim"

Practical Example

Let's have a look at a practical example where we will install the httpd and vsftpd services, enable them at boot time and have them auto restart after 5 seconds should they fail. When the httpd service is started, the vsftpd service should be started as well, but the httpd service should not fail if for some reason vsftpd can not be started.

[root@server1 ~]# dnf install httpd vsftpd -y
Last metadata expiration check: 0:02:49 ago on Fri 26 Jun 2020 12:36:40 +04.
Dependencies resolved.
.....
[root@server1 ~]# systemctl enable httpd && systemctl enable vsftpd
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/vsftpd.service → /usr/lib/systemd/system/vsftpd.service.
[root@server1 ~]# systemctl start  httpd && systemctl start vsftpd
[root@server1 ~]#

Now that we have both services installed, enabled at boot time and running, let's make sure they automatically restart in case they fail:

[root@server1 ~]# systemctl edit httpd
[root@server1 ~]# systemctl cat httpd
# /etc/systemd/system/httpd.service.d/override.conf
[Service]
Restart=on-failure
RestartSec=5s
[root@server1 ~]# systemctl daemon-reload

Apply the same configuration for the vsftpd service and test it:
Check for the Main PID of the service using the systemctl status command and send a SIGKILL signal. When checking the status again you should see Active: activating (auto-restart) and a few seconds later the service will be running again.

[root@server1 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/httpd.service.d
           └─override.conf
   Active: active (running) since Fri 2020-06-26 12:42:04 +04; 9min ago
     Docs: man:httpd.service(8)
 Main PID: 32234 (httpd)

[root@server1 ~]# kill -9 32234
[root@server1 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/httpd.service.d
           └─override.conf
   Active: activating (auto-restart) (Result: signal) since Fri 2020-06-26 12:51:33 +04; 2s ago
     Docs: man:httpd.service(8)
  Process: 32234 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=killed, signal=KILL)

Now, let's make sure that the vsftpd service is automatically started when httpd is started:

[root@server1 ~]# systemctl edit httpd
[root@server1 ~]# systemctl cat httpd
# /etc/systemd/system/httpd.service.d/override.conf
[Service]
Restart=on-failure
RestartSec=5s

[Unit]
Wants=vsftpd.service
[root@server1 ~]# systemctl daemon-reload

Stop both services then only start the httpd service again:

[root@server1 ~]# systemctl stop httpd && systemctl stop vsftpd
[root@server1 ~]# systemctl start httpd
[root@server1 ~]# systemctl status vsftpd
● vsftpd.service - Vsftpd ftp daemon
   Loaded: loaded (/usr/lib/systemd/system/vsftpd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/vsftpd.service.d
           └─override.conf
   Active: active (running) since Fri 2020-06-26 13:21:15 +04; 4s ago

Summary

We learned about different unit types in Systemd, the three sections of a Systemd unit file, understanding Target Units and managing units with the systemctl command.

An Introduction to Systemd and Systemd Unit files

Systemd Service Units

Systemd Mount Units

Systemd Socket Units

Systemd Target Units

Using Systemd to Manage Units

Managing Systemd Dependencies

Managing Unit Options

Practical Example

Summary

More from this blog

Kubernetes 101: Building Scalable Applications - ConfigMaps & Secrets

Kubernetes 101: Building Scalable Applications - Storage

Kubernetes 101: Building Scalable Applications - Networking

Kubernetes 101: Building Scalable Applications - Deployments

Kubernetes 101: Kubernetes Essentials

Command Palette

Systemd Service Units

Systemd Mount Units

Systemd Socket Units

Systemd Target Units

Using Systemd to Manage Units

Managing Systemd Dependencies

Managing Unit Options

Practical Example

Summary

More from this blog