---
title: "A service fills the disk with logs"
url: "https://axoflow.com/docs/axoflow/troubleshoot/log-flood/"
description: "Find the service that floods the logs, free disk space, rate-limit the service in the systemd journal, and cap the journal and syslog file size."
last_modified: "2026-10-09T11:07:28+02:00"
---

> For the complete documentation index, see [llms.txt](https://axoflow.com/docs/axoflow/llms.txt).

# A service fills the disk with logs

Find the service that floods the logs, free disk space, rate-limit the service in the systemd journal, and cap the journal and syslog file size.

If a service writes log messages faster than usual, the systemd journal or the plain syslog file can grow until the disk fills up. This page shows how to:

- find the service that writes the logs,
- free disk space,
- limit how many log messages the service can write to the journal,
- keep those messages out of the plain syslog file, and
- cap the total size of the journal so one service can’t fill the disk.

The examples use the [Cloud connector](https://axoflow.com/docs/axoflow/architecture/index.md#axoedge) on an [AxoRouter](https://axoflow.com/docs/axoflow/architecture/index.md#router) host. It runs as the `axoflow-cloudconnector.service` systemd unit. [Other AxoRouter services](https://axoflow.com/docs/axoflow/onboard-hosts/hosts/services/index.md) work the same way, for example, `axorouter.service`, `axostore.service`, or `axorouter-wec.service`. Set the `UNIT` and `PROGRAM` variables in the commands to the values of the service.

All commands need root privileges.

**CAUTION:**

Don’t edit the unit file of a service that an Axoflow package installs. For example, systemd generates `axoflow-cloudconnector.service` from the Podman Quadlet file `/etc/containers/systemd/axoflow-cloudconnector.container`, which the AxoRouter package installs. The next AxoRouter upgrade replaces this file and your change is lost. Use a systemd drop-in file instead: upgrades don’t touch drop-ins.

## Find the full filesystem and the noisy service

1. Check which filesystem is full, and how much space the logs use:

   ```bash
   df -h /var /run                       # which filesystem is full
   journalctl --disk-usage               # space used by the journal
   sudo du -xsh /var/log/* | sort -h     # space used by plain log files
   ```

   Check `/var/log/syslog` (Debian, Ubuntu) or `/var/log/messages` (RHEL, Rocky Linux) in particular.
2. Find the services that logged the most messages in the last hour:

   ```bash
   sudo journalctl --since -1h -o json --output-fields=_SYSTEMD_UNIT \
     | grep -o '"_SYSTEMD_UNIT":"[^"]*"' | sort | uniq -c | sort -rn | head
   ```
3. Check the recent messages of the service at the top of the list:

   ```bash
   UNIT=axoflow-cloudconnector.service
   journalctl -u "$UNIT" --since -1h | wc -l   # messages in the last hour
   journalctl -u "$UNIT" -n 50                 # recent messages
   ```

   The messages often show the cause of the log volume, for example, a connection error that repeats.

## Why the disk fills up

With its default settings, the journal limits its own size. It uses at most 10% of the filesystem (never more than 4 GiB), and leaves at least 15% of the filesystem free (never more than 4 GiB). If the disk is full anyway, the cause is usually one of these:

- **A plain-file copy of the logs.** rsyslog, syslog-ng, or AxoSyslog also writes the same messages to `/var/log/messages` or `/var/log/syslog`. This is the most common cause. The journal settings don’t limit the size of this copy. To fix it, [keep the service’s messages out of the syslog file](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md#exclude-from-syslog).
- **Increased journal size limits.** Someone raised `SystemMaxUse=` in `journald.conf`. To fix it, [cap the size of the journal](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md#limit-journal-size).
- **The service writes its own log file** instead of logging to stdout or stderr. The journal settings and the rate limit don’t apply to this file.
- **`Storage=volatile` is set in `journald.conf`.** journald keeps the journal in `/run` (RAM), so it fills up memory instead of the disk.

## Free disk space

If the disk is already full, run these commands to rotate the journal and shrink it to 500 MB.

**CAUTION:**

The `truncate` command empties the whole syslog file. This also deletes the logs of all other services on the host.

```bash
sudo journalctl --rotate                 # close the active journal files so that vacuum can remove them
sudo journalctl --vacuum-size=500M       # or: --vacuum-time=2d
# If the syslog file uses the space, empty it in place.
# Don't delete it with rm: the daemon keeps the deleted file open, and the space isn't freed.
sudo truncate -s 0 /var/log/syslog       # or /var/log/messages
```

## Rate-limit the logs of a service in the journal

The `LogRateLimitIntervalSec` and `LogRateLimitBurst` options of a systemd unit replace the global journald rate limit for that unit. The unit can write at most `LogRateLimitBurst` messages in each `LogRateLimitIntervalSec` interval. journald drops the excess messages.

1. Create a drop-in file that lets the unit write about 1000 messages every 30 seconds to the journal.

   **CAUTION:**

   The new limit only takes effect after a restart, which briefly interrupts the service. For the Axoflow Cloud Connector, collection from your cloud sources stops for a few seconds.

   ```bash
   UNIT=axoflow-cloudconnector.service
   sudo mkdir -p "/etc/systemd/system/$UNIT.d"
   sudo tee "/etc/systemd/system/$UNIT.d/50-log-ratelimit.conf" <<'EOF'
   [Service]
   # Keep at most 1000 messages per 30s from this unit; journald drops the rest
   LogRateLimitIntervalSec=30s
   LogRateLimitBurst=1000
   EOF
   sudo systemctl daemon-reload
   sudo systemctl restart "$UNIT"
   ```

   You can also create the drop-in file interactively with `sudo systemctl edit <service-name>`.
2. Check that the limit is active.

   ```bash
   UNIT=axoflow-cloudconnector.service
   systemctl status "$UNIT" | grep -A1 Drop-In   # lists 50-log-ratelimit.conf
   systemctl show "$UNIT" -p LogRateLimitIntervalUSec -p LogRateLimitBurst
   journalctl -u systemd-journald --since -10min | grep -i "suppressed.*$UNIT"
   ```

   The `systemctl show` command prints the values you set.

   - **If the values are empty,** the host runs systemd older than version 240, which doesn’t support these options. Check the version with `systemctl --version`. In that case, set the global `RateLimitIntervalSec=` and `RateLimitBurst=` options in `/etc/systemd/journald.conf.d/` instead (defaults: `30s` and `10000`). The global options apply to every service on the host.
   - **If the `grep` prints `Suppressed N messages` lines,** the limit is working and journald is dropping the excess messages. journald prints this line when a new 30-second window starts and the service is still logging, so wait at least a minute before checking.

   More messages can get through than the configured burst. journald raises the burst when the journal has a lot of space available. For example, with 4 GiB available, about 4000 messages get through in each 30-second window.

If you want to remove the rate limit, see [Remove the rate limit](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md#remove-rate-limit).

## Keep the service’s messages out of the syslog file

On Debian and Ubuntu, rsyslog receives messages from journald through the syslog socket. journald forwards these messages *before* it applies the rate limit. The [journald rate limit](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md#journal-rate-limit) therefore doesn’t reduce what rsyslog writes to `/var/log/syslog`. On RHEL and Rocky Linux, rsyslog reads the journal (`imjournal`), so it gets only the messages that pass the rate limit.

If rsyslog runs on the host (`systemctl is-active rsyslog`), add a rule that stops it from writing the messages of the service. The messages stay available in the journal, with the rate limit applied.

1. Find the program name of the service. rsyslog uses the `SYSLOG_IDENTIFIER` field of the journal as the program name. For the Axoflow Cloud Connector, it is `axoflow-cloudconnector`.

   ```bash
   UNIT=axoflow-cloudconnector.service
   journalctl -u "$UNIT" -n 1 -o verbose | grep SYSLOG_IDENTIFIER
   ```
2. Add the rule, and restart rsyslog.

   ```bash
   PROGRAM=axoflow-cloudconnector
   sudo tee "/etc/rsyslog.d/10-drop-$PROGRAM.conf" <<EOF
   # $PROGRAM logs to the systemd journal; keep it out of the syslog file
   :programname, isequal, "$PROGRAM" stop
   EOF
   sudo rsyslogd -N1                 # check the configuration
   sudo systemctl restart rsyslog
   ```

## Cap the size of the journal and the syslog file

Limiting the total size of the journal protects the disk, no matter which service produces the logs. Do this even if you already rate-limited the service.

```bash
sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/50-size.conf <<'EOF'
[Journal]
SystemMaxUse=2G
SystemKeepFree=4G
EOF
sudo systemctl restart systemd-journald
```

Also make sure that [`logrotate`](https://linux.die.net/man/8/logrotate) rotates the syslog file by size. Edit the existing entry for the syslog file in `/etc/logrotate.d/rsyslog` (Debian, Ubuntu) or `/etc/logrotate.d/syslog` (RHEL, Rocky Linux), and add the `maxsize` and `rotate` options to it. Don’t create a new file for the same log: logrotate reports an error for duplicate log entries. For example:

```text
/var/log/syslog
{
    maxsize 1G
    rotate 4
    # keep the other options of the existing entry
}
```

## Check that the disk usage stopped growing

Run these commands again an hour later, and compare the results:

```bash
journalctl --disk-usage
df -h /var
sudo du -xsh /var/log/* | sort -h
```

## Remove the rate limit

To remove the [rate limit of the service](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md#journal-rate-limit) and the [rsyslog rule](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md#exclude-from-syslog), run the following commands.

```bash
UNIT=axoflow-cloudconnector.service
PROGRAM=axoflow-cloudconnector
sudo rm "/etc/systemd/system/$UNIT.d/50-log-ratelimit.conf"
sudo systemctl daemon-reload
sudo systemctl restart "$UNIT"
# If you added the rsyslog rule:
sudo rm "/etc/rsyslog.d/10-drop-$PROGRAM.conf"
sudo systemctl restart rsyslog
```

## Limitations of the rate limit

- **Dropped messages are lost permanently.** They can include the error that explains the problem. Use the rate limit as a safety valve, not as a fix. The real fix is to remove the cause of the log volume, for example, to lower the log level of the service, or to upgrade AxoRouter. After that, you can raise or [remove the limit](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md#remove-rate-limit).
- **The limit only applies to messages that go through journald.** This includes stdout and stderr, `syslog()` calls, and containers that use the `journald` log driver. It doesn’t apply to a log file that the service writes itself.