systemd Restart Policies for a Home Server - Recover Services Without Hiding Failure

Written by Adam Muiz 11 Sep 2026 Updated: 11 Sep 2026 9 min read

A long-running service on a home server exits at 3 a.m. Should the machine leave it down, restart it immediately, or keep trying until morning? Automatic recovery sounds like the obvious answer, but an unlimited restart loop can turn one clear failure into noise, repeated load, and a harder diagnosis.

systemd provides several controls for this problem. The useful question is not simply "how do I make a service restart?" It is: which failures should trigger another attempt, how quickly, and when should the attempts stop? This article builds a conservative policy for a generic long-running service on Debian. It does not assume that one set of values fits every workload.

Recovery is not the same as repair

A restart can clear a transient state: a process may have encountered a temporary network error, exhausted a resource that was released on exit, or crashed because of an isolated defect. Starting a fresh process may restore service. It can also repeat a permanent error, such as an invalid configuration, a missing credential, an incompatible database schema, or a full disk.

This distinction matters because systemd supervises a process; it does not understand whether an application's responses are correct. A process can be running while returning errors, waiting forever on a dependency, or serving stale data. Conversely, a process that exits deliberately after finishing its work may be healthy even though it is no longer running.

Treat automatic restart as one layer in an operational system, not as proof of health. Logs, external checks, resource monitoring, and application-specific readiness still have jobs to do.

Choose which exits deserve another attempt

The central setting belongs in the unit's [Service] section. According to the Debian Trixie systemd.service manual, Restart=on-failure retries after events such as a non-zero exit code, an unclean signal, a timeout, or a watchdog failure. It does not normally restart after a clean exit or an explicit systemctl stop.

[Service]
Restart=on-failure
RestartSec=10s

For a long-running server process, on-failure is often a reasonable starting point because it distinguishes an error from an intentional clean exit. The upstream manual recommends it for long-running services, but that is guidance rather than a universal rule. An application may use a non-zero code to request operator intervention, or it may accidentally return zero after a fatal internal condition. The service's own exit contract remains important.

Restart=always is broader: it also restarts after a clean exit. That can suit a process that must remain present continuously and has no meaningful "finished successfully" state. It is not automatically more reliable. If an administrator or deployment script expects the program to exit cleanly and stay down, always works against that intent. There are also narrower modes such as on-abnormal and on-watchdog; choose them only when their documented exit categories match the application.

Exit codes can refine the policy

systemd also has SuccessExitStatus=, RestartPreventExitStatus=, and RestartForceExitStatus=. These can express an application's documented exit semantics. For example, a particular code could mean "configuration is invalid; do not retry." Adding such rules without an application contract is risky, however. A memorable number is not a substitute for documentation.

Bound both time and attempt count

RestartSec=10s inserts a delay before an automatic restart. An explicit delay prevents a failing process from being relaunched almost immediately. Ten seconds is only an example: a local DNS helper, a large Java service, and a worker calling a rate-limited remote API have different recovery costs.

systemd 254 introduced RestartSteps= and RestartMaxDelaySec=, which can increase the delay over several restart attempts. The checked Debian host runs systemd 257.13, so these directives are available there. Readers should check their own release with:

systemctl --version

A bounded increasing delay can be written as:

[Service]
Restart=on-failure
RestartSec=10s
RestartSteps=4
RestartMaxDelaySec=2min

The exact sequence is calculated by systemd between the initial and maximum delays; the key operational point is that attempts become less frequent and stop growing at the configured maximum. On releases older than 254, omit the two newer directives and use a fixed RestartSec=. Do not paste unknown directives and assume they were accepted: systemd may log and ignore an unknown setting.

Delay alone does not cap the number of starts. That job belongs to StartLimitIntervalSec= and StartLimitBurst= in the [Unit] section:

[Unit]
StartLimitIntervalSec=10min
StartLimitBurst=5

The systemd.unit manual defines this as a start-rate limit: more than the allowed number of starts within the interval is refused. It applies to all starts, including manual ones, not just those caused by Restart=. Failed unit condition checks do not count toward the limit.

When a restarting service reaches the limit, automatic attempts stop. This is not a permanent circuit breaker. After the interval has passed, a later manual, timer, or socket activation can start the unit and enable restart behavior again. This nuance is important if the desired policy is "remain down until a human approves recovery"; start-rate limiting alone does not provide that workflow.

Use a drop-in instead of editing a packaged unit

If a package owns the main unit file under /usr/lib/systemd/system/, editing it directly makes local policy easy to lose or confuse during upgrades. systemd supports drop-in files that are merged after the main unit. The systemctl manual documents systemctl edit as the interface for creating an override.

sudo systemctl edit example-worker.service

For a generic continuously running worker, the resulting drop-in could contain:

[Unit]
StartLimitIntervalSec=10min
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=10s
RestartSteps=4
RestartMaxDelaySec=2min

example-worker.service is a placeholder, not a service observed on this server. Replace it with the actual unit name and review that application's documentation before choosing values. Also inspect existing settings first:

systemctl cat example-worker.service
systemctl show example-worker.service \
  --property=Restart,RestartUSec,NRestarts,StartLimitIntervalUSec,StartLimitBurst

systemctl cat shows the main file and loaded drop-ins on disk. systemctl show presents normalized properties and runtime state; time properties commonly use a USec suffix even when the unit-file directive ends in Sec.

Verify before changing a running service

A configuration that looks tidy can still contain a misspelled directive, a misplaced section, or an invalid command. The systemd-analyze manual says that verify can detect unknown directives, missing required dependencies, missing documented manual pages, and commands that are absent or not executable.

systemd-analyze verify /path/to/example-worker.service

The complete generic example used for this article was checked with systemd 257.13 using an existing executable. That validates the unit syntax on that host; it does not prove that a real application will start, become ready, process requests correctly, or recover safely.

After reviewing and saving a real drop-in, systemctl edit reloads manager configuration when the editor exits successfully. Before restarting, inspect the merged result again. Then schedule the restart for a moment when interruption is acceptable:

sudo systemctl restart example-worker.service
systemctl status example-worker.service
journalctl --unit=example-worker.service --since today

A restart is itself a stop followed by a start, so it can interrupt active work. For a database, queue worker, or upload processor, the application's shutdown and resume behavior matters more than the elegance of the unit file.

When recovery stops, preserve the evidence

If the unit enters failed or reports a start-limit hit, begin with its status and journal rather than immediately clearing the state:

systemctl status example-worker.service
journalctl --unit=example-worker.service --since today

systemctl status is a human-readable view of current or recent runtime state and a small log excerpt. It is not a complete history. The journal can show earlier attempts, provided those records are retained.

Once the underlying problem has been identified and corrected, this command clears the failed state and the per-unit start and restart counters:

sudo systemctl reset-failed example-worker.service
sudo systemctl start example-worker.service

reset-failed does not fix a configuration, restore a credential, free disk space, or repair application data. It only clears systemd's recorded state and counters. Clearing it before collecting evidence can make the sequence of events harder to understand.

What automatic restart cannot guarantee

A process supervisor sees process state and the signals that an application exposes. It cannot, by itself, answer several larger questions:

Is the service returning correct responses rather than merely accepting connections?
Is a required database or remote API ready?
Did the previous process leave a partially completed job?
Will repeated starts increase load on a failing dependency?
Has anyone been alerted after automatic recovery stopped?

These are reasons to pair restart policy with observability and application design. A health check can look from the user's side. Structured logs can preserve the failure cause. Idempotent job handling can reduce damage from interrupted work. An alert can make a bounded retry policy actionable instead of silent.

There is also a counter-argument to aggressive recovery: for a non-essential home-server service, staying down may be safer than repeatedly touching storage, a remote API, or corrupted state. Availability is not the only objective. Resource use, data integrity, and diagnosability deserve explicit weight.

A practical review checklist

Confirm that the unit represents a long-running process, not a successful one-shot task.
Read the application's exit-code and shutdown documentation.
Choose on-failure, always, or a narrower mode deliberately.
Set an explicit restart delay appropriate to the service and its dependencies.
Add a start-rate limit, knowing that it also counts manual starts.
Check the installed systemd version before using newer directives.
Use a drop-in, inspect the merged unit, and run systemd-analyze verify.
Decide how a stopped recovery loop becomes visible to a person.
Test only in a safe window with a service whose failure behavior is understood.

Conclusion

A useful restart policy is not the one that keeps trying forever. It is the one that classifies expected failure, leaves enough time between attempts, stops before a loop becomes its own incident, and preserves a trail for diagnosis.

For many long-running services, Restart=on-failure, an explicit RestartSec=, and a considered start-rate limit form a defensible baseline. They remain only a baseline. The right values depend on what the application does when it starts, what it can damage when interrupted, and how an operator learns that recovery did not work.