how to write units with elegant reload

Developing a system with a graceful shutdown can be a bit of a dance with a tambourine. In an ideal world, each service would be managed by a unit systemd. ExecStart start a process that processes SIGTERMa ExecStop would notify the process and take a lock that gracefully terminates the process and its resources.

However, many programs end incorrectly, or even completely knock down all settings when closed. In this article, we will look at the behavior systemd at shutdown and methods for writing units systemd for selective cleaning (custom cleanup) before closing. Details – to the start of our DevOps course.

systemd

Like the system init, systemd manages services, along with many other tasks, from startup to shutdown. When loaded, it starts first, and when closed, it stops last. Unlike earlier sequential scripts Services systemd focused on units systemd. Their relationship is built on dependencies and streamlining. This allows you to start (or close) many services parallel. All this is important for the article in the future:

  • Services are started (and closed) in parallel, unless otherwise specified.

  • Processes terminate in SIGTERM or SIGKILL after a timeout, unless configured otherwise.

  • When shutting down, services with ordering dependencies are stopped in the reverse order they were started.

Shutdown

What happens at the end of a job? Several subordinate teams systemctl (subcommands, listed below) shut down the system, in doing so they activate special units systemd: reboot.target, poweroff.target and halt.target.

systemctl halt      # Завершение и остановка работы системы
systemctl poweroff  # Завершение работы системы и отключение питания
systemctl reboot    # Завершение работы системы и перезагрузка

Team Requires of these target units is requested to extract dependencies such as systemd-reboot.service, systemd-poweroff.serviceand systemd-halt.service (respectively), and those in reverse order request shutdown.target.

# reboot.target
[Unit]
Description=System Reboot
Documentation=man:systemd.special(7)
DefaultDependencies=no
Requires=systemd-reboot.service
After=systemd-reboot.service
AllowIsolate=yes
JobTimeoutSec=30min
JobTimeoutAction=reboot-force

[Install]
Alias=ctrl-alt-del.target
sudo systemctl list-dependencies --all --recursive reboot.target
reboot.target
○ └─systemd-reboot.service
●   ├─system.slice
●   │ └─-.slice
○   ├─final.target
○   ├─shutdown.target
○   └─umount.target

All units and scopes default to DefaultDependencies=yesthus, for them explicitly expressions added Before=shutdown.target and Conflicts=shutdown.target. Conflicts starts operation shutdown.target and stops conflicting units. launch shutdown.target simultaneously stops all conflicting units (unless otherwise specified).

When the unit stops working systemd must gracefully terminate running processes, release resources, and wait for the work to complete. The load balancer may stop accepting new connections and disable its readiness endpoint. The database can flush data to disk, and the agent can tell the cluster to leave the group. Any processes that continue to run after execution ExecStop terminated incorrectly (that is, forcibly) with the command SIGKILL in systemd (in the absence of other settings).

Many programs terminate incorrectly, or even completely knock down all their settings, deviate from the model systemd. Some shutdown tasks require coordination with the cluster system. Type tools systemd-analyze shutdown does not help. And many services may not even be your software. In these cases, cleanup actions in the early shutdown units can help. Consider some strategies for writing units systemdwhich perform user-specified cleanup actions before shutting down.

Cleanup script

Let’s start with a simple script cleanup, which simulates a cleaning task. Team echo logs messages, the loop shows progress, and sleep ensures that the task runs long enough to ensure that it waits for subsequent actions. Note that this script is independent of the network, containers, and other system components:

#!/bin/bash

echo "cleaning..."
for i in {1..3}; do
  sleep 5s
  echo "waiting..."
done

echo "done"

Let’s put this script in the directory /usr/local/bin/cleanup and make it executable. If you see an error later systemd 203/EXECreturn to this section.

Do not place this script in your home directory. The default SELinux policy does not allow devices systemd execute “home” scripts. And this is correct.

ExecStart Script

Now consider the systemd oneshot block, which runs cleanup like a script ExecStart. This will be our first and, as we shall soon see, a misguided attempt.

# /etc/systemd/system/clean.service
[Unit]
Description=Clean on shutdown
Before=shutdown.target    # implicit

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/cleanup

[Install]
WantedBy=shutdown.target

This unit creates a dependency shutdown.target from clean.serviceuses Before=shutdown.target when issuing a command clean.service before shutdown.targetuses Type=oneshotso that the service is considered running when the script exits (to delay shutting down the computer until the cleanup is complete). RemainAfterExit keeps the service active even after the script ends.

Default units systemd (Type=simple) are considered to be running at the same time as the process. This, for example, allows subsequent units to start working immediately.

Activate clean.service, systemctl rebootthen look at the operation history log …

Sep 28 23:55:01 ip-10-0-13-150 systemd[1]: Starting clean.service - Clean on shutdown...
Sep 28 23:55:01 ip-10-0-13-150 cleanup[6796]: cleaning...
-- Boot 8e4734a82c754e549c9a9292ca5988fb --

This is the wrong approach. We don’t even see history. The cleanup service may start, but this will not delay the shutdown. We have shutdown.target conflicts with all units, so clean.service interrupted before completion. could you add DefaultDependencies=nobut as we can read in systemd.unit man pagesit won’t delay shutdown either: “Given two units with any ordering dependency between them, if one unit is shut down and the other is started up, the shutdown is ordered before the start-up” , one of which starts its work, and the other completes it, the shutdown command will be executed earlier”).

ExecStop Script

Now consider the oneshot unit in systemdwhich starts the cleanup with a script ExecStop.

# /etc/systemd/system/clean.service
[Unit]
Description=Clean on shutdown
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecStop=/usr/local/bin/cleanup

[Install]
WantedBy=multi-user.target

This unit is being retrieved multi-user.targetbut in the process of launching it receives a command to execute After=multi-user.target quite late. Since command-controlled (ordered) units are closed in the reverse order of startup, the unit clean must to begin close before other units.

Activate and launch clean.service. We confirm that he active (even though we got out of it).

● .service - Clean on shutdown
     Loaded: loaded (/etc/systemd/system/clean.service; enabled; vendor preset: disabled)
     Active: active (exited) since Thu 2022-09-29 20:21:17 UTC; 2min 49s ago
    Process: 1383 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
   Main PID: 1383 (code=exited, status=0/SUCCESS)
        CPU: 2ms

Sep 29 20:21:17 ip-10-0-13-150 systemd[1]: Starting clean.service - Clean on shutdown...
Sep 29 20:21:17 ip-10-0-13-150 systemd[1]: Finished clean.service - Clean on shutdown.

Note the Boot ID in the activity log, then run systemctl reboot:

...
-- Boot 0e40d519972b4cd7bc09374b3072788d --
Sep 28 20:18:24 ip-10-0-13-150 systemd[1]: Starting clean.service - Clean on shutdown...
Sep 28 20:18:24 ip-10-0-13-150 systemd[1]: Finished clean.service - Clean on shutdown.

Let’s check the log after reboot. When was it launched systemctl rebootI indicated with a mark 🔁.

...
Sep 28 20:18:24 ip-10-0-13-150 systemd[1]: Finished clean.service - Clean on shutdown.
🔁
Sep 29 20:20:38 ip-10-0-13-150 systemd[1]: Stopping clean.service - Clean on shutdown...
Sep 29 20:20:38 ip-10-0-13-150 cleanup[367051]: cleaning...
Sep 29 20:20:43 ip-10-0-13-150 cleanup[367077]: waiting...
Sep 29 20:20:48 ip-10-0-13-150 cleanup[367080]: waiting...
Sep 29 20:20:53 ip-10-0-13-150 cleanup[367132]: waiting...
Sep 29 20:20:53 ip-10-0-13-150 cleanup[367134]: done
Sep 29 20:20:53 ip-10-0-13-150 systemd[1]: clean.service: Deactivated successfully.
Sep 29 20:20:53 ip-10-0-13-150 systemd[1]: Stopped clean.service - Clean on shutdown.
-- Boot 8e97b43271024c64b0775c43dc519c5b --
Sep 29 20:21:17 ip-10-0-13-150 systemd[1]: Starting clean.service - Clean on shutdown...
Sep 29 20:21:17 ip-10-0-13-150 systemd[1]: Finished clean.service - Clean on shutdown.

The unit stops working, but the script cleanup at this time, it works to the victorious, which delays the completion of the work. After that, a reboot occurs. On next boot, the unit will run (/bin/true), after which its work is completed.

Everything works as expected, but with one caveat. The work of command-controlled units is completed in the reverse order of launch. And here it is worth remembering how many services stop working at a time. Our script cleanup does not guarantee that certain services will work when executed in ExecStop. Himself cleanup works only because it does not depend on other components of the system. If it were, for example, dependent on the network, we would have to add After=network.targetso that when the work is completed, its work ends before the network services stop working.

Now let’s move on to the task of cleaning in containers, which is becoming commonplace these days.

Container

Consider the block systemdwhich starts a container process at ExecStop before shutdown. Unlike the minimalistic script we used earlier, running this container requires a lot of system components to run. ExecStop starts at the same time as other units stop working. It does not promise success.

# /etc/systemd/system/clean.service
[Unit]
Description=Clean on shutdown
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecStopPre=-/usr/bin/podman rm clean
ExecStop=/usr/bin/podman run \
  --name clean \
  --log-driver=k8s-file \
  --rm \
  -v /usr/local/bin:/scripts \
  --stop-timeout=60 \
  --entrypoint /scripts/cleanup \
  docker.io/fedora:36

[Install]
WantedBy=multi-user.target

Activate and run clean.service. With manual deactivation clean.service ExecStop will create a container, mount path /usr/local/bin and normally run the same cleanup.

Sep 29 21:36:41 ip-10-0-13-150 systemd[1]: Starting clean.service - Clean on shutdown...
Sep 29 21:36:41 ip-10-0-13-150 systemd[1]: Finished clean.service - Clean on shutdown.
Sep 29 21:37:35 ip-10-0-13-150 systemd[1]: Stopping clean.service - Clean on shutdown...
Sep 29 21:37:35 ip-10-0-13-150 podman[10701]: 2022-09-29 21:37:35.32232541 +0000 UTC m=+0.076846159 container create 2e0a21b085113ad6b5eab83a1f4b85081045727b711c89909dc7d204abb25e61 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 21:37:35 ip-10-0-13-150 podman[10701]: 2022-09-29 21:37:35.292987568 +0000 UTC m=+0.047508358 image pull  docker.io/fedora:36
Sep 29 21:37:35 ip-10-0-13-150 podman[10701]: 2022-09-29 21:37:35.516155501 +0000 UTC m=+0.270676209 container init 2e0a21b085113ad6b5eab83a1f4b85081045727b711c89909dc7d204abb25e61 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 21:37:35 ip-10-0-13-150 podman[10701]: 2022-09-29 21:37:35.5276157 +0000 UTC m=+0.282136425 container start 2e0a21b085113ad6b5eab83a1f4b85081045727b711c89909dc7d204abb25e61 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 21:37:35 ip-10-0-13-150 podman[10701]: 2022-09-29 21:37:35.527985526 +0000 UTC m=+0.282506267 container attach 2e0a21b085113ad6b5eab83a1f4b85081045727b711c89909dc7d204abb25e61 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 21:37:35 ip-10-0-13-150 podman[10701]: cleaning...
Sep 29 21:37:40 ip-10-0-13-150 podman[10701]: waiting...
Sep 29 21:37:45 ip-10-0-13-150 podman[10701]: waiting...
Sep 29 21:37:50 ip-10-0-13-150 podman[10701]: waiting...
Sep 29 21:37:50 ip-10-0-13-150 podman[10701]: done
Sep 29 21:37:50 ip-10-0-13-150 podman[10701]: 2022-09-29 21:37:50.536310798 +0000 UTC m=+15.290831522 container died 2e0a21b085113ad6b5eab83a1f4b85081045727b711c89909dc7d204abb25e61 (image=docker.io/library/fedora:36, name=clean, health_status=)
Sep 29 21:37:50 ip-10-0-13-150 podman[10873]: 2022-09-29 21:37:50.707597051 +0000 UTC m=+0.115808702 container remove 2e0a21b085113ad6b5eab83a1f4b85081045727b711c89909dc7d204abb25e61 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 21:37:50 ip-10-0-13-150 systemd[1]: clean.service: Deactivated successfully.
Sep 29 21:37:50 ip-10-0-13-150 systemd[1]: Stopped clean.service - Clean on shutdown.

But during the actual completion of the work ExecStop will not be able to create the container.

Sep 29 21:41:21 ip-10-0-13-150 systemd[1]: Stopping clean.service - Clean on shutdown...
Sep 29 21:41:21 ip-10-0-13-150 podman[12002]: 2022-09-29 21:41:21.263765577 +0000 UTC m=+0.192250288 container create 214a91497691b33a2ee77a0ad6dc3b3894e102abf84d641f5df2abfc842d1cd0 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 21:41:21 ip-10-0-13-150 podman[12002]: 2022-09-29 21:41:21.190543212 +0000 UTC m=+0.119027932 image pull  docker.io/fedora:36
Sep 29 21:41:21 ip-10-0-13-150 podman[12002]: time="2022-09-29T21:41:21Z" level=error msg="Unable to clean up network for container 214a91497691b33a2ee77a0ad6dc3b3894e102abf84d641f5df2abfc842d1cd0: \"error tearing down network namespace configuration for container 214a91497691b33a2ee77a0ad6dc3b3894e102abf84d641f5df2abfc842d1cd0: netavark: failed to delete if podman0: Received a netlink error message Operation not supported (os error 95)\""
Sep 29 21:41:21 ip-10-0-13-150 podman[12072]: 2022-09-29 21:41:21.689689685 +0000 UTC m=+0.185177533 container remove 214a91497691b33a2ee77a0ad6dc3b3894e102abf84d641f5df2abfc842d1cd0 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 21:41:21 ip-10-0-13-150 podman[12002]: Error: OCI runtime error: crun: sd-bus call: Transaction for libpod-214a91497691b33a2ee77a0ad6dc3b3894e102abf84d641f5df2abfc842d1cd0.scope/start is destructive (shutdown.target has 'start' job queued, but 'stop' is included in transaction).: Resource deadlock avoided
Sep 29 21:41:21 ip-10-0-13-150 systemd[1]: clean.service: Control process exited, code=exited, status=126/n/a
Sep 29 21:41:21 ip-10-0-13-150 systemd[1]: clean.service: Failed with result 'exit-code'.
Sep 29 21:41:21 ip-10-0-13-150 systemd[1]: Stopped clean.service - Clean on shutdown.
-- Boot 332c28360e38479c91f5cab4898413b4 --
Sep 29 21:41:45 ip-10-0-13-150 systemd[1]: Starting clean.service - Clean on shutdown...
Sep 29 21:41:45 ip-10-0-13-150 systemd[1]: Finished clean.service - Clean on shutdown.

I have made this mistake myself and have seen users struggle with it without realizing it. In today’s daemonless container engines, it’s difficult to define dependency ordering to ensure that a container can be reliably run during completion of work. This is an open problem area that depends on the container engine.

ExecStop on existing container

Now we have become smarter. If you can’t just take and start the container at the end of work, let the container be already running and waiting for a signal. Let’s create a new script that reflects this approach and call it await. Let’s run this script directly and make sure it’s waiting, and then on click Ctrl-C prints messages.

#!/bin/bash

cleanup() {
  echo "cleaning..."
  for i in {1..3}; do
    sleep 5s
    echo "waiting..."
  done

  echo "done"
}

trap cleanup SIGINT SIGTERM

echo "Awaiting signals"
sleep infinity & wait $!

Let’s make sure that the bash script await ran in a container as process #1 and handled signals. Non-builtin commands (like sleep) can delay signal processing in non-interactive contexts. Therefore, it is important to make sleep a background one and use the built-in (interruptible) command wait to wait for the previous command.

Create, activate and launch a new unit clean.service in systemd.

# /etc/systemd/system/clean.service
[Unit]
Description=Clean on shutdown
After=multi-user.target

[Service]
Type=simple
ExecStartPre=-/usr/bin/podman rm clean
ExecStart=/usr/bin/podman run \
  --name clean \
  --log-driver=k8s-file \
  --rm \
  -v /usr/local/bin:/scripts \
  --stop-timeout=60 \
  --entrypoint /scripts/await \
  docker.io/fedora:36
ExecStop=/usr/bin/podman stop clean
TimeoutStopSec=180

[Install]
WantedBy=multi-user.target

This is a typical unit with a long execution time and Type=simple. Podman starts the container and proxies signals (such as SIGTERM at shutdown) to the first process of the container. When a unit terminates, podman stops sending a signal SIGTERM the first process of the container (and SIGKILL after --stop-timeoutwhich takes 10 seconds by default).

As before, the unit is retrieved multi-user.targetbut receives a command to run After (after) multi-user.target, at a fairly late launch stage. Command-controlled (ordered) units terminate in the reverse order of startup, so termination for clean starts earlier than for other units. Service TimeoutStopSec determines how long systemd must wait for completion ExecStop before execution SIGKILL.

We confirm the activity of the unit, we wait for the signal.

systemctl status clean.service
● clean.service - Clean on shutdown
     Loaded: loaded (/etc/systemd/system/clean.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2022-10-21 16:34:41 UTC; 1min 12s ago
   Main PID: 239609 (podman)
      Tasks: 9 (limit: 4427)
     Memory: 18.6M
        CPU: 275ms
     CGroup: /system.slice/clean.service
             ├─ 239609 /usr/bin/podman run --name clean --log-driver=k8s-file --rm -v /usr/local/bin:/scripts --stop-timeout=60 --entrypoint /scripts/await docker.io/fedora:36
             └─ 239680 /usr/bin/conmon --api-version 1 -c 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 -u 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70/userdata -p /run/containers/storage/overlay-containers/21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70/userdata/pidfile -n clean --exit-dir /run/libpod/exits --full-attach -s -l k8s-file:/var/lib/containers/storage/overlay-containers/21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70/userdata/oci-log --conmon-pidfile /run/containers/storage/overlay-containers/21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70

Sep 29 16:34:41 ip-10-0-0-27 systemd[1]: Started clean.service - Clean on shutdown.
Sep 29 16:34:42 ip-10-0-0-27 podman[239609]: 2022-10-21 16:34:42.02827346 +0000 UTC m=+0.073162795 container create 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 16:34:42 ip-10-0-0-27 podman[239609]: 2022-10-21 16:34:41.999309821 +0000 UTC m=+0.044199164 image pull  docker.io/fedora:36
Sep 29 16:34:42 ip-10-0-0-27 podman[239609]: 2022-10-21 16:34:42.253726035 +0000 UTC m=+0.298615370 container init 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 16:34:42 ip-10-0-0-27 podman[239609]: 2022-10-21 16:34:42.263454996 +0000 UTC m=+0.308344331 container start 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 16:34:42 ip-10-0-0-27 podman[239609]: 2022-10-21 16:34:42.263872042 +0000 UTC m=+0.308761624 container attach 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 16:34:42 ip-10-0-0-27 podman[239609]: Awaiting signals

This approach has the advantage of fetching the image on startup rather than on shutdown. In this case, the running container can be checked.

Note the download ID and run systemctl reboot (marked with 🔁). After the reboot, check the activity log.

Sep 29 16:34:42 ip-10-0-0-27 podman[239609]: Awaiting signals
🔁
Sep 29 16:42:06 ip-10-0-0-27 podman[239609]: cleaning...
Sep 29 16:42:06 ip-10-0-0-27 podman[239609]: Terminated
Sep 29 16:42:06 ip-10-0-0-27 systemd[1]: Stopping clean.service - Clean on shutdown...
Sep 29 16:42:11 ip-10-0-0-27 podman[239609]: cleaning...
Sep 29 16:42:16 ip-10-0-0-27 podman[239609]: waiting...
Sep 29 16:42:21 ip-10-0-0-27 podman[239609]: waiting...
Sep 29 16:42:26 ip-10-0-0-27 podman[239609]: waiting...
Sep 29 16:42:26 ip-10-0-0-27 podman[239609]: done
Sep 29 16:42:26 ip-10-0-0-27 podman[239609]: waiting...
Sep 29 16:42:31 ip-10-0-0-27 podman[239609]: waiting...
Sep 29 16:42:36 ip-10-0-0-27 podman[239609]: waiting...
Sep 29 16:42:36 ip-10-0-0-27 podman[239609]: done
Sep 29 16:42:36 ip-10-0-0-27 podman[239609]: 2022-10-21 16:42:36.44559863 +0000 UTC m=+474.490487974 container died 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 (image=docker.io/library/fedora:36, name=clean, health_status=)
Sep 29 16:42:36 ip-10-0-0-27 podman[241961]: 2022-10-21 16:42:36.80223871 +0000 UTC m=+30.342432111 container cleanup 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 16:42:36 ip-10-0-0-27 podman[241961]: clean
Sep 29 16:42:36 ip-10-0-0-27 podman[239609]: 2022-10-21 16:42:36.907856847 +0000 UTC m=+474.952746174 container remove 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 16:42:36 ip-10-0-0-27 podman[239609]: time="2022-10-21T16:42:36Z" level=error msg="forwarding signal 15 to container 21009bcf34c2fbd3810f7301d1693dba084d9d098cdcbd9e3b15d5dc22e3bd70: container has already been removed"
Sep 29 16:42:36 ip-10-0-0-27 systemd[1]: clean.service: Deactivated successfully.
Sep 29 16:42:36 ip-10-0-0-27 systemd[1]: Stopped clean.service - Clean on shutdown.
-- Boot d0b5eba20ebd452aae5dbffaee19eff8 --

The cleanup task is running and is delaying shutdown, but the task is running twice. A running container process gets SIGTERM, and podman stop also sends SIGTERM. Let’s reproduce this when running the script await directly. Let’s press Ctrl-C many times – and we will see a multiple call to cleanup.

There are several ways to fix this. The first is removing the podman stop call from ExecStop. In this case, the hindrance is that during the normal operation of systemctl, operations such as systemctl stop clean.servicewill not work, and we will need to send to the container SIGTERM independently when debugging.

Let’s better clear or reset (clear/reset) this trap. Handling signals and concurrency in shell scripts is not very convenient, it would be much better in Go. But for now, let’s leave this decision.

cleanup() {
  trap - SIGINT SIGTERM
  echo "cleaning..."
  ...
}
...

Reload and restart clean.service. Pay attention to the Boot ID in the operation log, then try to start again systemctl reboot (marked with 🔁).

Sep 29 17:01:04 ip-10-0-0-27 podman[3688]: Awaiting signals
🔁
Sep 29 17:02:20 ip-10-0-0-27 podman[3688]: cleaning...
Sep 29 17:02:20 ip-10-0-0-27 podman[3688]: Terminated
Sep 29 17:02:20 ip-10-0-0-27 systemd[1]: Stopping clean.service - Clean on shutdown...
Sep 29 17:02:25 ip-10-0-0-27 podman[3688]: waiting...
Sep 29 17:02:30 ip-10-0-0-27 podman[3688]: waiting...
Sep 29 17:02:35 ip-10-0-0-27 podman[3688]: waiting...
Sep 29 17:02:35 ip-10-0-0-27 podman[3688]: done
Sep 29 17:02:35 ip-10-0-0-27 podman[3688]: 2022-10-21 17:02:35.391304416 +0000 UTC m=+91.032365358 container died 75079c6a180f82d398bb33e1e22a45bf4b281e84912e780842dd167658e6179f (image=docker.io/library/fedora:36, name=clean, health_status=)
Sep 29 17:02:35 ip-10-0-0-27 podman[4229]: 2022-10-21 17:02:35.740391215 +0000 UTC m=+0.325653570 container remove 75079c6a180f82d398bb33e1e22a45bf4b281e84912e780842dd167658e6179f (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 17:02:35 ip-10-0-0-27 podman[4090]: clean
Sep 29 17:02:35 ip-10-0-0-27 systemd[1]: clean.service: Main process exited, code=exited, status=143/n/a
Sep 29 17:02:35 ip-10-0-0-27 systemd[1]: clean.service: Failed with result 'exit-code'.
Sep 29 17:02:35 ip-10-0-0-27 systemd[1]: Stopped clean.service - Clean on shutdown.
-- Boot f8b835d07c7f497c83487bdf3bd3e319 --

Let’s make sure that cleanup launched at most once (3 posts waiting...), and that its completion has been delayed.

There is another possible fix. Exit code 143 indicates exiting the container due to a graceful shutdown request SIGTERM; systemd marks it as a mistake, we would rather call it a success. Set service status SuccessExitStatus.

SuccessExitStatus=143
Sep 29 17:09:56 ip-10-0-0-27 podman[4981]: Awaiting signals
🔁
Sep 29 17:16:03 ip-10-0-0-27 podman[4981]: cleaning...
Sep 29 17:16:03 ip-10-0-0-27 podman[4981]: Terminated
Sep 29 17:16:03 ip-10-0-0-27 systemd[1]: Stopping clean.service - Clean on shutdown...
Sep 29 17:16:08 ip-10-0-0-27 podman[4981]: waiting...
Sep 29 17:16:13 ip-10-0-0-27 podman[4981]: waiting...
Sep 29 17:16:18 ip-10-0-0-27 podman[4981]: waiting...
Sep 29 17:16:18 ip-10-0-0-27 podman[4981]: done
Sep 29 17:16:18 ip-10-0-0-27 podman[4981]: 2022-10-21 17:16:18.636792684 +0000 UTC m=+382.160266981 container died 91888ef150ea54a9831736a966302d1811028ce3216afb4368bf0a266e61ee52 (image=docker.io/library/fedora:36, name=clean, health_status=)
Sep 29 17:16:18 ip-10-0-0-27 podman[7028]: 2022-10-21 17:16:18.968155309 +0000 UTC m=+15.157225407 container cleanup 91888ef150ea54a9831736a966302d1811028ce3216afb4368bf0a266e61ee52 (image=docker.io/library/fedora:36, name=clean, health_status=, maintainer=Clement Verna <cverna@fedoraproject.org>)
Sep 29 17:16:18 ip-10-0-0-27 podman[7028]: clean
Sep 29 17:16:19 ip-10-0-0-27 systemd[1]: clean.service: Deactivated successfully.
Sep 29 17:16:19 ip-10-0-0-27 systemd[1]: Stopped clean.service - Clean on shutdown.
-- Boot 3517a0a13b3e4885be9901666c0fd173 --

What’s next?

As we have seen, completing work with the help of a unit systemd – a delicate matter. But we paid attention only to the early shutdown phase, and not to those services (or operations) that should stop working as late as possible (for example, event logs). However, we have considered several possible strategies.

AT next post we will apply these methods to the Kubernetes Kubelet. A Kubelet service registers with a Kubernetes cluster and launches containers via a container runtime with features such as hooks preStop, terminationGracePeriod and disruption budgets. Stopping the Kubelet service does not notify the cluster or stop the containers, nor did (until recently) disable any of them.

Want more content like this?

  • Stay tuned for new blog tweets @poseidonlabs

  • Support Poseidon’s open source work by becoming one of our sponsors

  • Chat with us on email tech@psdn.io

Source

The examples discussed in this article are available at blog-bits licensed under MPL 2.0. They were tested on Fedora CoreOS 36.20221001.3.0 (systemd 250.8).

[Unit]
Description=Clean on shutdown
After=multi-user.target

[Service]
Type=simple
ExecStartPre=-/usr/bin/podman rm clean
ExecStart=/usr/bin/podman run \
  --name clean \
  --log-driver=k8s-file \
  --rm \
  -v /usr/local/bin:/scripts \
  --stop-timeout=60 \
  --entrypoint /scripts/await \
  docker.io/fedora:36
ExecStop=/usr/bin/podman stop clean
TimeoutStopSec=180
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target
#!/bin/bash

cleanup() {
  trap - SIGINT SIGTERM
  echo "cleaning..."
  for i in {1..3}; do
    sleep 5s
    echo "waiting..."
  done

  echo "done"
}

trap cleanup SIGINT SIGTERM

echo "Awaiting signals"
sleep infinity & wait $!

If you find a bug, please send your fix and I’ll try to update the code.

Links to resources

And we will teach you how to work with Linux so that you can upgrade your career or become a sought-after IT specialist:

To view all courses, click on the banner:

Brief catalog of courses

Data Science and Machine Learning

Python, web development

Mobile development

Java and C#

From basics to depth

As well as

88

Similar Posts

Leave a Reply Cancel reply