systemd Service Watchdogs - Detect a Process That Is Alive but No Longer Progressing

Written by Adam Muiz 13 Oct 2026 Updated: 13 Oct 2026 9 min read

A process can still have a PID, consume little CPU, and appear as active (running) while its useful work has stopped. Its event loop may be blocked, a worker may be waiting forever, or one internal lock may prevent every request from progressing. A restart rule based only on process exit cannot react, because the process has not exited.

A systemd service watchdog offers a narrower signal: a cooperating application promises to send periodic keep-alive messages. If the messages stop, systemd treats the missed deadline as a failure. The important word is cooperating. Adding one directive does not teach an arbitrary daemon how to report health, and a heartbeat is not proof that users are receiving correct responses.

The useful question is therefore not simply, “How do I enable WatchdogSec=?” It is: what progress does each heartbeat represent, and what should happen when that evidence disappears?

Running, responsive, and correct are different states

Process supervision can answer whether a process started, exited, received a signal, or exceeded a configured timeout. That already catches many failures. It does not automatically know whether the main loop is moving or whether the application is producing correct results.

A service watchdog adds an application-to-manager signal. According to the versioned Debian systemd.service(5) manual, WatchdogSec= configures a deadline after startup completes. The service must regularly notify systemd with WATCHDOG=1. If too much time passes between notifications, systemd marks the unit failed and terminates it with SIGABRT by default; WatchdogSignal= can select another signal.

This is stronger than “the PID exists,” but weaker than an end-to-end check. A program can send a heartbeat while returning HTTP 500 responses, reading stale data, or failing to reach a database. The watchdog only knows that the code path responsible for its message ran before the deadline.

The watchdog is a two-sided contract

The unit file supplies the deadline. systemd then exposes that deadline to the service in WATCHDOG_USEC, measured in microseconds, and may provide WATCHDOG_PID to identify the expected process. The sd_watchdog_enabled(3) API lets a daemon check these values instead of hard-coding an interval.

On the return path, the application sends notifications through the socket identified by NOTIFY_SOCKET. The sd_notify(3) protocol documentation defines WATCHDOG=1 as the keep-alive message that updates systemd's watchdog timestamp. It also defines READY=1, which reports completed startup for a Type=notify service. Readiness is a transition; a watchdog heartbeat is recurring evidence.

Upstream recommends sending the keep-alive every half of the watchdog interval. For a 30-second deadline, that implies aiming near every 15 seconds rather than waiting until the last moment. This margin helps with ordinary scheduling delay, but it does not make 30 seconds a universally good value. The deadline must leave room for the longest healthy operation on that specific application's path.

Heartbeat placement defines what is being watched

Consider two designs. In the first, a separate timer thread sends WATCHDOG=1 regardless of what the worker is doing. The process may continue to look alive even when its request loop is permanently blocked. In the second, the main loop sends a heartbeat only after it completes a meaningful iteration or verifies that its workers have advanced.

The second design provides more useful evidence, but “meaningful” is application-specific. A queue worker might require the consumer loop to poll successfully. A web server might check that its event loop is turning and that worker coordination is responsive. A heartbeat should not perform an expensive full diagnostic on every interval, yet it should not be detached from all useful work either.

This is a design interpretation built on the notification protocol, not a guarantee supplied by systemd. PID 1 cannot inspect application intent. The daemon author must choose which internal progress allows the ping.

Notification attribution matters too. NotifyAccess= controls which service processes may send accepted messages. When WatchdogSec= is used without an explicit value, systemd implicitly uses NotifyAccess=main. The protocol documentation warns that short-lived helper processes can be difficult to attribute unless access and synchronization are designed correctly. More importantly, a generic external helper that keeps pinging while the real application is stuck defeats the purpose of the watchdog.

A conservative unit shape

For software that explicitly supports systemd readiness and watchdog notifications, the relevant part of a unit might look like this:

[Service]
Type=notify
ExecStart=/usr/local/bin/example-worker
WatchdogSec=30s
Restart=on-failure
RestartSec=10s

example-worker is a placeholder, not a service observed on this server. This configuration is appropriate only if the executable sends READY=1 after initialization and then sends attributable WATCHDOG=1 messages. Without READY=1, a Type=notify service does not complete its normal startup transition. Without recurring watchdog messages, it will fail after the configured deadline.

Restart=on-failure includes watchdog failures. Restart=on-watchdog is narrower if only a missed watchdog should cause an automatic restart. Neither choice is harmless by definition. Restarting can interrupt requests, repeat non-idempotent work, or add load to an unhealthy dependency. Automatic attempts are also subject to systemd's start-rate limits.

The existing restart policy, shutdown behavior, persistent state, and recovery contract still need review. A watchdog detects one class of failure; it does not decide whether replaying the process is safe.

Inspect support before changing the unit

Start with the application's own documentation and packaged unit. Search for explicit support for sd_notify, WatchdogSec, WATCHDOG_USEC, or a documented systemd notification option. The systemd daemon(7) integration guide treats status notification as behavior implemented by a modern daemon, not something inferred by the manager.

systemctl cat example-worker.service
systemctl show example-worker.service \
  --property=Type,NotifyAccess,WatchdogUSec,WatchdogTimestamp,Result

systemctl cat reveals the packaged unit and loaded drop-ins on disk. systemctl show exposes normalized manager properties; unit-file settings ending in Sec often appear as properties ending in USec. A zero watchdog interval means service watchdog supervision is disabled.

If a shell-based service intentionally implements the protocol, systemd-notify(1) is the command-line wrapper around sd_notify(). The watchdog message is an ordinary assignment:

systemd-notify WATCHDOG=1

That command is not a recommendation to add a cron job or external loop. It must run inside the supervised service context, with correct notification access, at a point that represents real application progress.

Validate configuration, then test behavior safely

A unit can be checked before installation:

systemd-analyze verify ./example-worker.service

The systemd-analyze(1) manual says verify can report problems such as unknown directives, missing required dependencies, and unavailable commands. A clean result validates part of the configuration. It cannot prove that the application sends pings, that the ping follows meaningful work, or that restarting preserves data.

Behavioral testing should use a non-production instance or a maintenance window. Observe a normal startup first, including readiness and several heartbeat intervals. Then use the application's supported test mechanism to pause or block the monitored loop without corrupting state. Expected evidence includes a watchdog failure in the journal, the configured termination signal, the unit's Result, and any restart allowed by its policy.

systemctl status example-worker.service
journalctl --unit=example-worker.service --since today
systemctl show example-worker.service \
  --property=ActiveState,SubState,Result,NRestarts,WatchdogTimestamp

Do not test by freezing an unfamiliar production database, storage service, or queue worker. A forced failure can hold locks, interrupt writes, trigger client retries, or produce a core dump. The test plan needs an explicit stop condition and a rollback that removes the drop-in or restores the previous unit settings.

What a service watchdog cannot prove

It cannot prove that an HTTP response is correct or even successful.
It cannot prove that a database, DNS resolver, remote API, or storage device is healthy.
It cannot detect a logical bug if the heartbeat path continues to run.
It cannot guarantee that terminating and restarting the process is safe.
It cannot replace monitoring from the network or user side.

An external health check and an internal watchdog answer different questions. The external check asks whether a client can obtain an acceptable result. The watchdog asks whether selected code inside the service has reported progress within its deadline. Either can fail while the other passes. Using both can narrow uncertainty, but neither should be described as complete proof of health.

There are also services for which a watchdog adds little value. A short-lived job already bounded by a runtime timeout, an application with no trustworthy progress point, or software that cannot safely restart may be better served by logs, alerts, and a deliberate operator response. More automatic recovery is not always more reliable.

A cautious adoption checklist

Confirm that the exact application and version support systemd watchdog notifications.
Define which useful progress must occur before a heartbeat is sent.
Choose a deadline longer than valid worst-case work, with scheduling margin.
Review NotifyAccess= and ensure the expected process sends the message.
Decide whether watchdog failure should restart the service and whether that restart is safe.
Keep start-rate limits, logs, and an alert path so repeated failure does not become silent churn.
Verify unit syntax and test a missed heartbeat away from production traffic.
Retain an external check for behavior the internal heartbeat cannot observe.

The heartbeat should mean something

A systemd service watchdog fills a real gap between “the process exited” and “the process still exists.” Its value comes from a small, explicit contract: the manager sets a deadline, and the application reports continued progress before that deadline expires.

The mechanism becomes misleading when the ping is treated as a ritual. If an unrelated helper can send it forever, the green signal outlives the work it was supposed to represent. Start with the application's failure model, place the heartbeat behind meaningful progress, and regard a restart as a recovery attempt rather than a repair. The result is not perfect knowledge of service health, but it is more honest evidence than a PID alone.