Home Server & Self-Hosting

systemd Journal Retention on a Home Server - Keep Logs Useful Without Losing the Disk

systemd Journal Retention on a Home Server - Keep Logs Useful Without Losing the Disk

How long should a home server keep its logs? The tempting answer is “as long as possible”. Logs can explain why a service stopped, which boot introduced a problem, or whether a configuration change had an effect. But a journal that grows without a deliberate storage policy eventually competes with the applications it is meant to help.

systemd-journald already has retention controls. The difficult part is not finding a command that deletes old entries. It is understanding what is being stored, which limit is active, and what evidence should remain before anything is removed. This article focuses on that reasoning rather than proposing one universal number of days.

What the systemd journal stores

systemd-journald collects messages from several sources, including the kernel, services, the syslog API, and service standard output and error. The records are structured, so a query can filter by a unit, boot, priority, or other journal field instead of treating every line as unlabelled text. The service documentation describes the journal as a set of indexed records, not merely a text file that happens to be written under /var/log.

Storage can be persistent or volatile. Persistent journal files normally live below /var/log/journal. Volatile files normally live below /run/log/journal and may disappear when the machine reboots. The exact behavior also depends on Storage= and whether the persistent directory exists. A server that loses its journal at every reboot has a very different diagnostic history from one that keeps previous boots.

The official journald.conf documentation also explains an important transition: early in boot, data may be collected in volatile storage and later flushed to persistent storage when the conditions for persistent logging are met. That is why “I configured persistent logging” and “every message was already on disk” should not be treated as the same statement.

Inspect before choosing a limit

Start with observation. These commands do not change the retention policy:

sudo journalctl --disk-usage
sudo journalctl --list-boots
sudo journalctl --no-pager -p warning..alert -b 0
sudo journalctl --verify

--disk-usage reports the space used by active and archived journal files. --list-boots shows which boot history is available. The priority query is only an example of narrowing a diagnosis; a warning is not automatically an incident, and the absence of a warning does not prove that a service is healthy. --verify checks the internal consistency of journal files. It is a diagnostic check, not a guarantee that the messages are complete or truthful.

For a service-specific view, use the unit name rather than scanning every message:

sudo journalctl --no-pager --unit=nginx.service --since="today"
sudo journalctl --no-pager --boot=-1 --unit=nginx.service

The journalctl manual documents --unit, --boot, time filters, and the distinction between the system and user journals. Replace nginx.service with a unit that actually exists on the machine. A command copied from an article should not be mistaken for evidence about a particular server.

Three retention questions, not one

How much space may the journal use?

SystemMaxUse= limits journal storage on a persistent file system. RuntimeMaxUse= applies to volatile storage. The values are not the only constraints: SystemKeepFree= and RuntimeKeepFree= reserve space for other uses, and journald respects the smaller effective allowance. The documented defaults are percentage-based and capped, but a distribution build or local configuration can change the result.

This is a storage budget, not a promise that the journal will occupy exactly that amount. Only journal files with the expected suffixes are considered, and active files cannot always be removed immediately. A disk-usage report after cleanup can therefore differ from the configured maximum.

How old may an entry be?

MaxRetentionSec= provides a time-based upper bound. It is disabled by default in the documented configuration, which does not mean that logs are unlimited: size and free-space limits can still cause older archived files to be removed. A time limit can be useful when a project has a clear diagnostic horizon, but “14 days” is not inherently safer or more correct than “90 days”. The right value depends on how quickly problems are discovered, how much storage is available, and whether another system receives copies of the logs.

How large and how numerous may files become?

SystemMaxFileSize= and RuntimeMaxFileSize= influence rotation granularity. SystemMaxFiles= and RuntimeMaxFiles= limit the number of individual files. These controls help shape how history is divided, but they do not turn the journal into an archive. The Debian systemd-journald.service manual describes archived files as files that are no longer being written; that distinction matters when interpreting cleanup results.

A conservative drop-in

Local overrides are better kept in a drop-in than mixed into a vendor-provided main file. The upstream documentation lists /etc/systemd/journald.conf.d/*.conf as a supported location and describes the precedence rules. For example, an administrator could create a policy like this after deciding that the values fit the machine:

# /etc/systemd/journald.conf.d/60-retention.conf
[Journal]
SystemMaxUse=1G
SystemKeepFree=2G
MaxRetentionSec=30day

The numbers above are examples, not recommendations for every home server. A machine with a small root file system may need less. A machine used for incident investigation may need more or may need forwarding to a separate system. The point of a drop-in is to make the decision visible and reviewable.

After writing a configuration, review the file carefully and restart the journal service using the normal service manager. A restart is not a license to delete evidence:

sudo systemctl restart systemd-journald.service
sudo journalctl --disk-usage

Read the resulting usage and check that the services you care about still produce logs. The service documentation notes that restarting journald through a restart operation preserves service stream connections; stopping it separately is a different and less desirable operation for normal maintenance.

Manual vacuuming and the active-file trap

When a disk is already under pressure, journalctl offers explicit cleanup operations:

sudo journalctl --rotate
sudo journalctl --vacuum-time=30days
sudo journalctl --disk-usage

--vacuum-time removes archived files containing entries older than the requested period. The related options --vacuum-size= and --vacuum-files= apply size and file-count criteria. The official manual makes a caveat that is easy to miss: vacuuming operates on archived files, while --disk-usage includes active files. Rotating first makes more data eligible for cleanup, but it still does not guarantee that the final usage will equal the target.

Do not use manual vacuuming as a substitute for backups. Deleting a journal can remove useful evidence, and it cannot restore a database, application upload, or configuration. If logs matter after a machine failure, copy them to a separate destination before applying an aggressive policy. For sensitive environments, also consider who can read the journal and whether log forwarding creates a second place that needs access control.

Retention is also an information policy

More history is not always more useful. A noisy service can fill the journal with repetitive messages, while rate limiting can drop messages during a burst. The RateLimitIntervalSec= and RateLimitBurst= settings exist for that reason, but changing them without understanding the application can exchange disk pressure for missing records.

There is also a privacy dimension. Logs may contain IP addresses, paths, usernames, tokens accidentally printed by an application, or request data. Keeping everything forever increases the amount of information that must be protected. A retention policy should therefore answer two questions together: how long is this information useful, and how long is it appropriate to keep?

For a small server, a reasonable process is modest: measure current usage, identify the logs needed for troubleshooting, set a documented budget, test the configuration, and review it after a real incident or major service change. The documentation gives us the mechanics, but it cannot decide what evidence a particular project needs.

Conclusion

Keeping systemd journal logs is not simply a choice between “never delete” and “delete everything older than a week”. Persistent and volatile storage answer different questions. Size, free-space, time, file-size, and file-count settings constrain different parts of the system. Manual vacuuming works on archived files, so active files explain why cleanup may appear incomplete.

The safest starting point is inspection, followed by a small and explicit policy. If a log is important enough to keep beyond the server's storage budget, it may be important enough to copy elsewhere. That leaves an open operational question for every home server: which records would you genuinely need after the next failure, and where would they exist if this disk stopped working?

References