Introduction to Systemd - Manage your services on Linux with Systemd

We will speak today about systemd, the main init and system manager across several Linux distribution.

what is Systemd ?

Maybe you were familiar and used with initd (SysVinit) or Upstart to manage your service on Linux, but it was replaced in 2011 for the first time on Fedora distribution and very fast adopted by all the major distributions like RedHat, Ubuntu, Debian…

Basically, SystemD is the manager of everything on Linux which started just after the kernel booting.

As a reminder, here is the Linux booting process:

• 1)System startup and BIOS (Hardware)
• 2)Boot loader Stage 1 (MBR loading)
• 3)Boot loader Stage 2 (GRUB loader)
• 4)Kernel
• 5)INIT/SystemD -> Target

💭 As SystemD is the first process to started after the kernel, it will always have the first PID (Process ID).

Target, Unit and Services

SystemD is like an orchestrator for all Unit, Service and Target.

• the unit is a configuration file that will describe a target, a service, a process and how to manage it.

• a service is simply a daemon that can be started, stopped, restarted,…

Service is the most known Unit but there are a lot of other different unit, as by example, Devices, Mount, Sockets, Timer, Slice, Target,…

• A target is like a defined state to reach, it’s a group of units which can be managed as one.

It was previously known as run-levels.

The following table provides a brief description of few run-levels with equivalent target-units in the systemd program.

If you want to view your default boot target, use the following command :

 systemctl get-default

To change the default target :

# systemctl set-default [target]
systemctl set-default multiuser.target

Advantages of SystemD in comparison with System V init

The old SystemV init was starting all services in a pre-defined sequence.

It executes the next script in the sequence only if the current script in the sequence is executed or timed out.

If a script stuck during the execution, it had to wait until that script timed out.

This unexpected wait makes the entire system initialization process less efficient and ultimately slower.

SystemD in comparison starts services in parallel mode and then is faster and more efficient.

Systemd makes the boot process much simpler, entirely removing the need to specify dependencies.

Systemd can handle the boot process from start to end, without needing to use any of the existing shell scripts. Cool !

Moreover, the systemd Unit file are declarative and quite easy to configure and maintain.

Great! We have a winner ! 🏆

Manage your service with SystemD

There is an amazing command-line tool to interact with systemD on your OS, it’s called Systemctl . It will be used to query and manage your systemd state in real time.

Usage :

systemctl [options] (command) [unit ...]

List your Units and get their state

# List units currently in memory ( same output )
systemctl (no arguments)
systemctl list-units

# Filter the output of list units information
# output as well all the service system loaded or try to loaded 
systemctl list-units --all 

# Output only unit service
systemctl list-units --type=service

# Only the unit inactive
systemctl list-units --all --state=inactive

# Only the unit running
systemctl list-units --all --state=running

# List Installed Unit files configuration 
systemctl list-unit-files [query]

# Check if the service is active ( running )
systemctl is-active application.service

# Check if the service is enable 
systemctl is-enabled application.service

# Manage state of the unit 
systemctl status (unit)
systemctl status sshd.service

Start and stop your service with systemctl

In the old days, you were certainly using the ‘service’ command or ‘chkconfig’, as in the examples below:

service sshd restart

In few OS, it will still work but now, it’s time to use systemctl command.

We are SystemCtl !

# Start a service
systemctl start (unit)
# Stop a service
systemctl stop (unit)

# brutal stop - if service is stucked
systemctl kill (unit)

#After a change in your running service, restart or reload
systemctl restart (unit)
# reload the unit configuration file without restarting
systemctl reload (unit)

🤔 Keep in mind that when you are using the reload subcommand, the service is re-reading the configuration file and there is no interruption.

When you are using restart command, all current connection will be terminated and the configuration file will be re-read.

Reload causes less downtime, but not all services support it unhappily.

💭 notice then all Service unit ends with .service, then to manage your ‘service’, you need to add the suffix ‘.service’.

systemctl status rsyslog.service
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-01-27 09:01:00 UTC; 5min ago

However, almost all commands will still work without using the suffix ‘.service’ but it’s recommended to use it.

systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-01-27 09:01:00 UTC; 5min ago

Enable or disable service

If you start a service but it’s not enabled, it will not be persistent and the service will not started after the next reboot.

To make a service enabled, meaning it will started automatically after each reboot, you will need to use this systemctl subcommand.

To start a service at boot, use the enable command:

# enable the service from starting automatically
systemctl enable unit

Keep in mind that enabling a service does not start it in the current session, to do it you need to add the flag ‘–now’

# enable the service and start it 
systemctl enable --now unit

To see if the unit is enabled, you can use the is-enabled command:

# enable the service and start it 
systemctl is-enabled unit

To disable a service to start automatically :

# enable the service and start it 
systemctl disable unit

Same here, disable an Unit will not stop it if it’s running, but there is no flag for it, you need to use the stop command.

# enable the service and start it 
systemctl stop unit
systemctl disable unit

Change the state of your OS using systemd

As systemD is the master of the system and all the service running, you can use as well systemctl to control the state of your OS.

# reboot the system
systemctl reboot

# Poweroff/halt
systemctl poweroff 
systemctl halt

# change to specific target ( will start or stop service follow your target)
systemctl (target)
systemctl emergency
systemctl default

Systemctl has a lot of subcommands and if you are new on it, it’s hard in the beginning to remember, a little trick is to type systemctl, add a space and hit TAB key twice to display all available systemctl subcommands and to use the flag –help.

systemctl --help

How to manage your Unit config file ?

All your unit files are written in 3 different location:

• /etc/systemd/system: administrator environment, highest priority, contains the target configuration
• /run/systemd/: non-persistent ( at the run level ), info of the units running
• /usr/lib/systemd/system: package provided, lowest priority ( ex: sshd.service )

interesting

You can show the content of your file in 2 ways : detailed

• using the cat command directly on the file

cat /usr/lib/systemd/system/sshd.service

• Using systemctl, no need to know where is the file, faster and better

 systemctl cat sshd.service

While the specific format for unit files is outside of the scope for this first tutorial, systemctl provides built-in mechanisms for editing and modifying unit files if you need to make adjustments.

To edit, you can do directly with vim or your favorite editor or you can use systemctl edit, it will make a drop-in in /etc/systemd/system/servicename.d/, that you will be able easily to see the change you did and you will be able to easily rollback.

This will be a blank file that can be used to override or add directives to the unit definition.

systemctl edit sshd.service

💡 strangely, systemctl in some OS is using nano as editor for the unit, to change it set export SYSTEMD_EDITOR=vim in /etc/bashrc.

To rollback:

rm -r /etc/systemd/system/nameunit.service.d
rm -r /etc/systemd/system/sshd.service.d

# reload the daemon 
sudo systemctl daemon-reload

If you wish to edit the full unit file, you can pass the –full flag:

systemctl edit --full sshd.service

Troubleshooting and logs

One big advantage with SystemD is that all system unit logs are captured on the systemd journal and you can easily troubleshoot if there is any issue on any service using the command ‘journalctl’.

⚠️ The logs for journalctl are by default not persistent and will be lost after any reboot. If you want to make it persistent, you need to change the configuration file of journald.

sed -i 's/#Storage.*/Storage=persistent/' /etc/systemd/journald.conf
systemctl restart systemd-journald.service

⚠️ The restart of the systemd-journald service will drop the current log ( stored in /run/log/journal/)

Then, your log will be stored in directory /var/log/journal/ .

To follow the current log:

journalctl -f 

If you want to analyze the logs of the kernel ( kernel ring buffer ):

journalctl --dmsg
journalctl -k 

To see only log message related to a service/unit or few services:

journalctl -u sshd.service
journalctl -u sshd -u rsyslog

To limit message follow the time:

journalctl --since "yesterday"
journalctl --since "20 min ago"

Filter by type of priority or criticality:

journalctl -p err
journalctl -p crit
journalctl -p emerg

Analyses systemd booting performance

A really good command which is part of systemd is the systemd-analyze one.

It will give you an overview of the booting time for each of your unit to reach the defined target.

systemd-analyze
Startup finished in 1.965s (kernel) + 4.616s (initrd) + 14.834s (userspace) = 21.416s
multi-user.target reached after 10.110s in userspace

If you want to have more information for each service:

systemd-analyze blame
          5.061s kdump.service
          3.627s cloud-init-local.service
          2.243s cloud-init.service
          1.314s initrd-switch-root.service
          1.284s NetworkManager-wait-online.service
          1.246s cloud-final.service
          1.244s cloud-config.service
          1.045s tuned.service

You can as well ‘Analyses’ your service unit for security :

systemd-analyze security sshd

Conclusion

By now, hopefully, you should know the basics of how to manage a server that uses systemd. There is still a lot to know and if I can give you an advice, it’s to experiment and to use it a lot to understand the power of systemd and the feature of all these amazing command.

Hope you enjoy this introduction ! See you in the next episode !