Log Management for Home Server — Collecting Logs from journald, rsyslog, to Loki

There was a moment where a service on the home server suddenly went down, but I didn't know where to start looking. Open a terminal, type systemctl status, then remember that the log might be in /var/log, maybe in journalctl, or even buried in a deleted Docker container. It was like looking for house keys in the middle of the night with a flashlight with a low battery: every corner held possibilities, but nothing was certain.

This is where log management comes in. Not just keeping error records, but compiling them into a system that can be read, searched and relied on when something goes wrong. This article is a record of my journey in compiling logs on the home server, from the simplest method using journald, then to rsyslog, until finally playing with Loki and Grafana.

Why Logs Are More Important Than They Seem

Logs are digital traces of every decision the system makes. The application says "I received this request", the kernel says "there is a hardware oddity", the firewall says "I rejected the packet from this IP". Without logs, we only see the final symptoms: service down, website 500, slow connection. With logs, we can trace the cause-and-effect chain to the root of the problem.

I like to analogize logs to medical records. It is impossible for a doctor to diagnose just by looking at a patient coughing; he needs history, examination results, and trends over time. Logs provide that history to our system. The neater the medical records, the faster we know what's wrong.

The problem is, on a home server that runs many services, logs can be scattered in many places: text files in /var/log, systemd output in journald, application logs in containers, or even output from custom scripts that we write ourselves. Without a strategy, searching for a single incident can take hours.

journald — First Log from systemd

If you use a modern Linux distribution such as Debian, Ubuntu, or Fedora, systemd already handles many processes. Each unit that systemd runs — whether service, timer, or socket — will record its standard output to journald. This is the first central log that is almost always available.

The command I use most often is this:

# Lihat log suatu service secara real-time
journalctl -u nginx.service -f

# Lihat log dari boot terakhir
journalctl -b

# Lihat log dalam rentang waktu tertentu
journalctl --since "1 hour ago"

# Cari log yang mengandung kata tertentu
journalctl -u myapp.service | grep "error"

One of the strengths of journald is built-in structured logging. Each log message is accompanied by metadata such as microsecond precision timestamps, systemd units, PIDs, and even some additional fields from the application itself. The search is richer than grep a plain text file.

But journald also has limitations. By default, logs are saved in binary format in /var/log/journal. Although efficient, this format cannot be opened with regular cat or tail. Additionally, logs are only stored on one machine. If we have several servers, we have to log in one by one to look for problems.

For a single machine home server, journald is quite reliable. I usually set log retention so the disk doesn't get full:

# Edit konfigurasi journald
sudo nano /etc/systemd/journald.conf

# Tambahkan atau ubah baris berikut
[System]
Storage=persistent
MaxFileSec=1week
SystemMaxUse=500M
SystemMaxFiles=5

The above configuration ensures that the log remains on disk, but not more than 500 MB. The log files are rotated weekly, so we don't have to worry about /var/log bloating silently.

rsyslog — Classic Logs That Are Still Reliable

Before journald came along, rsyslog was the king of logs in the Linux world. Until now, it is still widely used because it is flexible and based on text files. Many legacy applications and networking devices — such as routers, switches, or access points — still send logs in syslog format to UDP port 514.

On the home server, I usually leave rsyslog running to capture logs from the devices. A simple configuration to receive logs from the local network is like this:

# File: /etc/rsyslog.conf atau /etc/rsyslog.d/10-server.conf
module(load="imudp")
input(type="imudp" port="514")

$template RemoteLogs,"/var/log/remote/%fromhost-ip%/%programname%.log"
if $fromhost-ip != '127.0.0.1' then ?RemoteLogs
& stop

With that configuration, each device that sends syslog logs will have its own folder in /var/log/remote/, filled with files according to the program name. This is very helpful when I want to view router or access point logs without having to open the web admin.

The advantage of text files is their universality. We can open it with tail, grep, awk, or even tools like lnav. No need for a special database. But on the other hand, text files are difficult to search on a large scale. Imagine looking for a single error line among millions of separate log lines across dozens of files.

Loki — Centralized Logging for Modern Home Servers

As the number of services started to increase, I felt I needed a more convenient place to search for logs. The solution is Loki, a log aggregation system from Grafana Labs. Loki is designed to be light, resource efficient, and suitable for home servers.

Unlike Elastic Stack which stores logs as full JSON documents, Loki only indexes labels. The contents of the log are still stored in a format similar to object storage. As a result, Loki is much more RAM and CPU efficient, something that is important if your home server only uses a used laptop like mine.

The simplest setup can be using Docker Compose like this:

# File: docker-compose.yml
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml:ro
    command: -config.file=/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  loki-data:
  grafana-data:

Promtail is an agent that reads logs from files and then sends them to Loki. We just say to Promtail: "read this file, label it". Later in Grafana, we can filter logs based on labels such as job, container, or host.

Example Promtail configuration for reading Docker logs and syslog:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          __path__: /var/log/syslog

  - job_name: docker
    static_configs:
      - targets:
          - localhost
        labels:
          job: docker
          __path__: /var/lib/docker/containers/*/*.log

In Grafana, the query is similar to Prometheus but for logs. For example, searching for all logs containing the word error from a Docker container:

{job="docker"} |= "error"

Or look for logs from a particular systemd service that has been forwarded to file:

{job="syslog"} |= "nginx" |= "500"

The syntax is intuitive. We can chain filters, regexes, and even extract fields from logs with the pattern or regexp parser. This is much more convenient than opening dozens of terminal tabs.

Simple Architecture that You Can Implement

If you're just starting out, there's no need to install everything straight away. I myself carry out evolution like this:

• Stage 1 — One machine: Rely on journalctl and /var/log. Enough for learning and debugging simple problems.
• Stage 2 — Several services: Enable rsyslog to capture logs from network devices and applications that still use the classic syslog format.
• Stage 3 — Multiple containers or VMs: Install Loki + Promtail + Grafana so all logs can be searched from one dashboard.

What matters is not how sophisticated the tools are, but how consistently the logs are collected. The application you just deployed must have a log strategy from day one. Don't wait for the first incident to think about logs.

For applications I write myself, I usually structure the logs in a consistent format. A simple example of a Python application:

import logging
import sys

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
    handlers=[
        logging.StreamHandler(sys.stdout),
        logging.FileHandler("/var/log/myapp/app.log")
    ]
)

logger = logging.getLogger("myapp")
logger.info("Server started on port 3001")

The output to stdout will be captured by journald if the application is running as a systemd service. The output to the file will be read by Promtail and entered into Loki. That way, the logs have redundancy without me having to bother.

Best Practices That Can Be Applied Immediately

After playing with logs for a while, there are a few habits I've learned and recommend:

• Use UTC or a consistent time zone. If logs from several servers have different time zones, finding the sequence of events becomes difficult. We recommend that all logs use UTC, or at least note the offset.
• Do not log sensitive data. Passwords, tokens, and personal information should not appear in logs. Logs are files that are opened frequently and can be stored for a long time.
• Set log rotation. Either use logrotate for text files or configure retention in journald and Loki. Full disk due to logs is a very preventable problem.
• Give meaningful labels. In Loki, labels are the key to search. Use labels such as environment=production, service=api, or host=server-lima.
• Monitor logs as well as metrics. Don't just install Loki but never open it. Create a simple dashboard in Grafana to see the number of errors per hour or response time anomalies.

Configuration of logrotate for custom applications can be like this:

# File: /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0644 www-data www-data
}

Daily rotation with 14 backup files means we keep logs for two weeks. Sufficient for home server needs while not burdening storage.

Conclusion

Log management on a home server is not about installing the most advanced tools. It's about having visibility. When something goes wrong, we don't need to panic looking for traces because everything is neatly arranged. From the ever-present journald, the reliable rsyslog for classic devices, to the Loki that makes log searching convenient — every tool has its role.

Start small. Make sure every service you run has a log that can be read. Adjust the rotation so that the disk does not fill up. If you feel you need a faster search, then consider Loki. Most importantly, don't wait for a problem to come before you realize that the log that was supposed to be there was never saved.

If you have your own experience with log management on a home server, write it in the comments column. I'm curious how you organize your logs and what tools you use most often. Don't forget to share this article if you find it useful.