A service fills the disk with logs

If a service writes log messages faster than usual, the systemd journal or the plain syslog file can grow until the disk fills up. This page shows how to:

• find the service that writes the logs,
• free disk space,
• limit how many log messages the service can write to the journal,
• keep those messages out of the plain syslog file, and
• cap the total size of the journal so one service can’t fill the disk.

The examples use the Cloud connector on an AxoRouter host. It runs as the axoflow-cloudconnector.service systemd unit. Other AxoRouter services work the same way, for example, axorouter.service, axostore.service, or axorouter-wec.service. Set the UNIT and PROGRAM variables in the commands to the values of the service.

All commands need root privileges.

Find the full filesystem and the noisy service

1. Check which filesystem is full, and how much space the logs use:

   df -h /var /run                       # which filesystem is full
   journalctl --disk-usage               # space used by the journal
   sudo du -xsh /var/log/* | sort -h     # space used by plain log files

   Check /var/log/syslog (Debian, Ubuntu) or /var/log/messages (RHEL, Rocky Linux) in particular.

2. Find the services that logged the most messages in the last hour:

   sudo journalctl --since -1h -o json --output-fields=_SYSTEMD_UNIT \
     | grep -o '"_SYSTEMD_UNIT":"[^"]*"' | sort | uniq -c | sort -rn | head

3. Check the recent messages of the service at the top of the list:

   UNIT=axoflow-cloudconnector.service
   journalctl -u "$UNIT" --since -1h | wc -l   # messages in the last hour
   journalctl -u "$UNIT" -n 50                 # recent messages

   The messages often show the cause of the log volume, for example, a connection error that repeats.

Why the disk fills up

With its default settings, the journal limits its own size. It uses at most 10% of the filesystem (never more than 4 GiB), and leaves at least 15% of the filesystem free (never more than 4 GiB). If the disk is full anyway, the cause is usually one of these:

• A plain-file copy of the logs. rsyslog, syslog-ng, or AxoSyslog also writes the same messages to /var/log/messages or /var/log/syslog. This is the most common cause. The journal settings don’t limit the size of this copy. To fix it, keep the service’s messages out of the syslog file.
• Increased journal size limits. Someone raised SystemMaxUse= in journald.conf. To fix it, cap the size of the journal.
• The service writes its own log file instead of logging to stdout or stderr. The journal settings and the rate limit don’t apply to this file.
• Storage=volatile is set in journald.conf. journald keeps the journal in /run (RAM), so it fills up memory instead of the disk.

Free disk space

If the disk is already full, run these commands to rotate the journal and shrink it to 500 MB.

sudo journalctl --rotate                 # close the active journal files so that vacuum can remove them
sudo journalctl --vacuum-size=500M       # or: --vacuum-time=2d
# If the syslog file uses the space, empty it in place.
# Don't delete it with rm: the daemon keeps the deleted file open, and the space isn't freed.
sudo truncate -s 0 /var/log/syslog       # or /var/log/messages

Rate-limit the logs of a service in the journal

The LogRateLimitIntervalSec and LogRateLimitBurst options of a systemd unit replace the global journald rate limit for that unit. The unit can write at most LogRateLimitBurst messages in each LogRateLimitIntervalSec interval. journald drops the excess messages.

1. Create a drop-in file that lets the unit write about 1000 messages every 30 seconds to the journal.

   CAUTION:

   The new limit only takes effect after a restart, which briefly interrupts the service. For the Axoflow Cloud Connector, collection from your cloud sources stops for a few seconds.

   UNIT=axoflow-cloudconnector.service
   sudo mkdir -p "/etc/systemd/system/$UNIT.d"
   sudo tee "/etc/systemd/system/$UNIT.d/50-log-ratelimit.conf" <<'EOF'
   [Service]
   # Keep at most 1000 messages per 30s from this unit; journald drops the rest
   LogRateLimitIntervalSec=30s
   LogRateLimitBurst=1000
   EOF
   sudo systemctl daemon-reload
   sudo systemctl restart "$UNIT"

   You can also create the drop-in file interactively with sudo systemctl edit <service-name>.

2. Check that the limit is active.

   UNIT=axoflow-cloudconnector.service
   systemctl status "$UNIT" | grep -A1 Drop-In   # lists 50-log-ratelimit.conf
   systemctl show "$UNIT" -p LogRateLimitIntervalUSec -p LogRateLimitBurst
   journalctl -u systemd-journald --since -10min | grep -i "suppressed.*$UNIT"

   The systemctl show command prints the values you set.

   • If the values are empty, the host runs systemd older than version 240, which doesn’t support these options. Check the version with systemctl --version. In that case, set the global RateLimitIntervalSec= and RateLimitBurst= options in /etc/systemd/journald.conf.d/ instead (defaults: 30s and 10000). The global options apply to every service on the host.
   • If the grep prints Suppressed N messages lines, the limit is working and journald is dropping the excess messages. journald prints this line when a new 30-second window starts and the service is still logging, so wait at least a minute before checking.

   More messages can get through than the configured burst. journald raises the burst when the journal has a lot of space available. For example, with 4 GiB available, about 4000 messages get through in each 30-second window.

If you want to remove the rate limit, see Remove the rate limit.

Keep the service’s messages out of the syslog file

On Debian and Ubuntu, rsyslog receives messages from journald through the syslog socket. journald forwards these messages before it applies the rate limit. The journald rate limit therefore doesn’t reduce what rsyslog writes to /var/log/syslog. On RHEL and Rocky Linux, rsyslog reads the journal (imjournal), so it gets only the messages that pass the rate limit.

If rsyslog runs on the host (systemctl is-active rsyslog), add a rule that stops it from writing the messages of the service. The messages stay available in the journal, with the rate limit applied.

1. Find the program name of the service. rsyslog uses the SYSLOG_IDENTIFIER field of the journal as the program name. For the Axoflow Cloud Connector, it is axoflow-cloudconnector.

   UNIT=axoflow-cloudconnector.service
   journalctl -u "$UNIT" -n 1 -o verbose | grep SYSLOG_IDENTIFIER

2. Add the rule, and restart rsyslog.

   PROGRAM=axoflow-cloudconnector
   sudo tee "/etc/rsyslog.d/10-drop-$PROGRAM.conf" <<EOF
   # $PROGRAM logs to the systemd journal; keep it out of the syslog file
   :programname, isequal, "$PROGRAM" stop
   EOF
   sudo rsyslogd -N1                 # check the configuration
   sudo systemctl restart rsyslog

Cap the size of the journal and the syslog file

Limiting the total size of the journal protects the disk, no matter which service produces the logs. Do this even if you already rate-limited the service.

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/50-size.conf <<'EOF'
[Journal]
SystemMaxUse=2G
SystemKeepFree=4G
EOF
sudo systemctl restart systemd-journald

Also make sure that logrotate rotates the syslog file by size. Edit the existing entry for the syslog file in /etc/logrotate.d/rsyslog (Debian, Ubuntu) or /etc/logrotate.d/syslog (RHEL, Rocky Linux), and add the maxsize and rotate options to it. Don’t create a new file for the same log: logrotate reports an error for duplicate log entries. For example:

/var/log/syslog
{
    maxsize 1G
    rotate 4
    # keep the other options of the existing entry
}

Check that the disk usage stopped growing

Run these commands again an hour later, and compare the results:

journalctl --disk-usage
df -h /var
sudo du -xsh /var/log/* | sort -h

Remove the rate limit

To remove the rate limit of the service and the rsyslog rule, run the following commands.

UNIT=axoflow-cloudconnector.service
PROGRAM=axoflow-cloudconnector
sudo rm "/etc/systemd/system/$UNIT.d/50-log-ratelimit.conf"
sudo systemctl daemon-reload
sudo systemctl restart "$UNIT"
# If you added the rsyslog rule:
sudo rm "/etc/rsyslog.d/10-drop-$PROGRAM.conf"
sudo systemctl restart rsyslog

Limitations of the rate limit

• Dropped messages are lost permanently. They can include the error that explains the problem. Use the rate limit as a safety valve, not as a fix. The real fix is to remove the cause of the log volume, for example, to lower the log level of the service, or to upgrade AxoRouter. After that, you can raise or remove the limit.
• The limit only applies to messages that go through journald. This includes stdout and stderr, syslog() calls, and containers that use the journald log driver. It doesn’t apply to a log file that the service writes itself.