Log rotation and journald configuration

Topic: Servers linux

Prevent logs from filling the disk: configure logrotate for app logs and limit journald size (SystemMaxUse, RuntimeMaxUse). Verify rotation runs and disk usage stays bounded so the system does not crash from a full disk.

Intent: How-to

Quick answer

  • logrotate: add config in /etc/logrotate.d/ for your app (rotate daily/weekly, keep N copies, compress); run logrotate -d /path to test; cron runs logrotate daily.
  • journald: set SystemMaxUse= and RuntimeMaxUse= in /etc/systemd/journald.conf; restart journald; journalctl --vacuum-size= to shrink now.
  • Verify: ls -la /var/log after rotation; df -h /var; ensure rotation is in cron and logrotate -f runs without error.

Prerequisites

Steps

  1. Configure logrotate for app logs

    Create /etc/logrotate.d/myapp: path to log file, daily/weekly, rotate 7, compress, missingok; run logrotate -d /etc/logrotate.d/myapp to dry-run.

  2. Limit journald size

    Edit /etc/systemd/journald.conf: SystemMaxUse=500M, RuntimeMaxUse=200M; systemctl restart systemd-journald; journalctl --vacuum-size=100M to shrink immediately.

  3. Ensure cron runs logrotate

    Check /etc/cron.daily/logrotate or systemd timer; run logrotate -f /etc/logrotate.conf to force once; check /var/log for rotated .gz files.

  4. Verify and monitor

    df -h /var; du -sh /var/log; list rotated files; set alert when /var usage exceeds threshold.

Summary

You will set up log rotation for application logs and cap journald size so logs do not fill the disk. Use this on every server that writes logs and after fixing a disk-full caused by logs.

Prerequisites

  • Root or sudo; knowledge of where the app writes logs.

Steps

Step 1: Configure logrotate for app logs

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
}

Test: sudo logrotate -d /etc/logrotate.d/myapp

Step 2: Limit journald size

sudo sed -i 's/^#*SystemMaxUse=.*/SystemMaxUse=500M/' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald
sudo journalctl --vacuum-size=100M

Step 3: Ensure cron runs logrotate

ls /etc/cron.daily/logrotate
sudo logrotate -f /etc/logrotate.conf
ls -la /var/log/*.gz

Step 4: Verify and monitor

  • df -h and du -sh /var/log; confirm rotated files appear; add monitoring for /var usage.

Verification

  • After a day (or after logrotate -f), rotated logs exist and journal size is under the limit; disk usage stable.

Troubleshooting

Logrotate did not rotate — Check config syntax; ensure log file path exists and is written by the app; run with -v and fix errors.

Journal still large — Restart journald after config change; run vacuum-size or vacuum-time; check for a different journal path (e.g. persistent storage).

Next steps

Continue to