Home Server & Self-Hosting

systemd OnFailure Hooks for a Home Server - Route Failure Without Calling It Monitoring

Written by Adam Muiz Updated: 05 Nov 2026 8 min read
systemd OnFailure Hooks for a Home Server - Route Failure Without Calling It Monitoring

A service fails at 02:00, leaves a useful explanation in the journal, and stays unnoticed until someone needs it the next morning. A restart policy may recover the process, but recovery and visibility are different questions. How can a small home server create an explicit path from “this unit failed” to “record or report this event” without pretending that the path is a complete monitoring system?

systemd provides a compact building block for that path: OnFailure=. It does not send email or call a webhook by itself. According to the official systemd.unit documentation, it activates one or more named units when the source unit enters the failed state. That precise boundary is what makes the feature useful, and also what prevents it from being a universal alerting solution.

Start with the state transition, not the notification channel

A systemd unit may be active, inactive, failed, or in a transitional state. The systemctl manual describes failed as similar to inactive, except that the unit failed in some way, such as an error exit, crash, timeout, or too many restart attempts. Exact causes still depend on the unit type and its configuration.

OnFailure= responds to that state. A clean systemctl stop normally makes a service inactive rather than failed, so it is not an alert test. A unit skipped because a Condition...= check was not met also did not necessarily fail. This distinction matters: alerting on every inactive unit would turn normal shutdowns and intentionally dormant services into noise.

It helps to separate three jobs:

  • Detection: systemd decides that the source unit entered failed.
  • Routing: OnFailure= asks systemd to activate a handler unit.
  • Delivery: the handler records the event locally or passes it to an external channel.

Only the first two are provided by OnFailure=. A mail server, chat API, or webhook can still be unreachable. Keeping those layers separate makes failures easier to reason about.

Attach a reusable handler with a drop-in

Suppose a service is named example-app.service. Instead of editing a package-supplied unit file, add a drop-in at /etc/systemd/system/example-app.service.d/failure-hook.conf:

[Unit]
OnFailure=unit-failure@%n.service

The %n specifier expands to the full source unit name. For this example, systemd activates an instance corresponding to [email protected]. The repeated .service looks odd, but the first occurrence belongs to the instance value and the last is the handler unit's own suffix.

Now create the template /etc/systemd/system/[email protected]:

[Unit]
Description=Record failure of %i

[Service]
Type=oneshot
ExecStart=/usr/bin/logger --tag=unit-failure Failure hook activated for %i

Inside the template, %i expands to the instance name, which is example-app.service in this case. The example deliberately uses logger rather than a real remote notifier. It proves that the routing path runs and leaves a local breadcrumb. It does not expose credentials, depend on a particular provider, or imply that a remote message was delivered.

A production handler can replace that command with a small, narrowly scoped notifier. Its interface should accept a unit name, derive a short message, apply a timeout, and return a meaningful exit status. Secrets should not be pasted into a public unit example or embedded directly in a command line. The handler also should not run as root merely because the source service does; give it only the identity, files, and network access its delivery method needs.

Verify syntax before asking PID 1 to use it

Reloading and testing should be deliberate. The official systemd-analyze verify documentation says the command can report unknown directives, missing required dependencies, unavailable commands, and several other structural errors.

sudo systemd-analyze verify \
  example-app.service \
  /etc/systemd/system/[email protected]
sudo systemctl daemon-reload
systemctl cat example-app.service

If the source unit comes from a package, verify the effective unit and its drop-ins with systemctl cat. A clean verification result is useful, but limited: it cannot prove that DNS works, an API token is valid, a recipient exists, or a remote service accepted the message. Syntax validation and delivery testing are separate gates.

Test with a disposable failure

Breaking a real web server to test an alert is an unnecessary gamble. A temporary test unit can exercise the state transition without touching an application. Save this as /etc/systemd/system/failure-hook-test.service and give it the same hook:

[Unit]
Description=Disposable failure-hook test
OnFailure=unit-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/bin/false

After another daemon-reload, start the test and inspect both sides:

sudo systemctl start failure-hook-test.service
systemctl status failure-hook-test.service --no-pager
systemctl status [email protected] --no-pager
journalctl -u failure-hook-test.service --no-pager
journalctl -u [email protected] --no-pager
journalctl -t unit-failure --no-pager

The first command is expected to return a failure because /usr/bin/false exits unsuccessfully. That non-zero result is part of the test, not evidence that the hook was skipped. The source status and journal should explain the source failure; the handler status and tagged journal entry should show whether the secondary unit ran. The journalctl -u documentation notes that this filter includes messages emitted by and about the selected unit.

After recording the result, remove the disposable unit, reload the manager, and clear its remembered failure:

sudo systemctl reset-failed failure-hook-test.service
sudo rm /etc/systemd/system/failure-hook-test.service
sudo systemctl daemon-reload

reset-failed clears the failed state and also resets per-unit start-limit and restart counters. It is cleanup, not diagnosis; read the journal before erasing useful runtime state.

Restart policy and failure routing are different controls

Restart=on-failure asks whether systemd should try to run a service again. OnFailure= asks whether another unit should be activated when the source reaches failed. A service may recover after one restart and still deserve a record, while a permanently failing service may eventually hit its start limit. The systemd.service documentation describes both restart causes and rate limiting; neither setting should be used to hide an unresolved failure loop.

There is also a version-sensitive exception. RestartMode=direct, added in systemd 254, skips the failed/inactive transition during automatic restart and therefore skips OnFailure= for those attempts. Debian bookworm's systemd 252 documentation does not include that newer mode, while current upstream documentation does. Check systemctl --version and the effective service configuration rather than assuming every host follows the newest manual.

Design for the handler to fail too

A handler lives on the same machine as the problem it reports. If the disk is full, name resolution is broken, the network is down, or the host loses power, it may not deliver anything. A local journal entry is still valuable for later diagnosis, but it cannot tell someone about a machine that is completely unreachable. External uptime checks observe from a different failure domain and remain necessary when absence itself must be detected.

Repeated crashes introduce a different problem: too many messages. systemd start-rate limits can bound service restart attempts, but a notifier may still need its own deduplication or cooldown policy. A useful event should identify the host, unit, time, and a small stable result field; it should link or point to logs rather than copy an unlimited journal into every message. Avoid including environment variables or command lines blindly because they may contain credentials.

The handler's own unit must also be observable. Check it with systemctl --failed, inspect its journal, and test the real delivery channel periodically. A green source service does not prove last month's notifier credentials still work. Conversely, a failed notifier should not restart or stop the application it is meant to report; the reporting path should remain a side channel, not a new hard dependency.

Use the hook for a narrow promise

OnFailure= is a small mechanism with a useful contract: when systemd records a real failed-state transition, it can activate another unit. A per-service drop-in, a reusable template, a local journal breadcrumb, and a disposable test are enough to make that contract visible.

The honest promise stops there. The hook does not prove an application is healthy, detect a powered-off host, guarantee a webhook arrived, or decide which failures deserve waking someone. Treat it as one link in an observability chain: precise enough to route a local failure, simple enough to test, and limited enough that external monitoring still has a clear job.

References