April 17, 2025

and

Send syslog data to Grafana Loki with syslog-ng

Grafana Loki is a popular tool to store syslog and other log data, because of its powerful search interface based on metadata labels and the dashboards you can create in Grafana. With AxoSyslog (our binary compatible, drop-in syslog-ng™ replacement) you can send data directly to Loki, making it easy to collect and route your data. AxoSyslog also makes it easy to add labels based on the content of the log messages, making it an ideal log collector and processing tool to feed Grafana Loki. (Don’t worry if you haven’t used AxoSyslog yet, it’s easy to install, or you can upgrade your existing syslog-ng™ installations in minutes.)

Install Loki and Grafana

First, you have to install Loki and Grafana. For this blog we'll use a local deployment using the official repositories, for other deployment methods (like Helm) see the official installation docs of Loki and Grafana.

The command examples refer to AlmaLinux, you might have to adjust them slightly for other distributions.

Add the Grafana repository to your package manager

Create a repository file:

sudo tee /etc/yum.repos.d/grafana.repo <<< '[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt' > /dev/null

Deploy Loki and Grafana

Update your package lists:

sudo dnf update

Then install Loki and Grafana:

sudo dnf install loki grafana-enterprise

Start the Grafana server:

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Make Grafana start automatically (in case you want to use this host more than once):

sudo systemctl enable grafana-server.service

If you’re installing Grafana and Loki on a VM/external machince and want to connect from an external host, you must enable port 3000/tcp to access Grafana, and 9096/tcp to send data to Loki via gRPC in the firewall/iptables rules of the host.

Connect Loki to Grafana

Add a Loki data source to make the data you send to Loki available from Grafana.

Log in to Grafana: http://localhost:3000/ (the default username/password is admin/admin)
Select Connections > Data Sources > Add data source > Loki.
Configure how Grafana communicates with Loki. For this demo (since both services are running on the same host), it's OK to use No Authentication, so set URL to http://localhost:3100 and Authentication method to No Authentication.

Click Save & test.

By now you should have a Loki deployment connected to a Grafana deployment, so if you send some logs to Loki, it will show up in Grafana. Next, we'll install AxoSyslog and get it to send logs to Loki.

Install AxoSyslog

To keep things simple, we'll install AxoSyslog on the same AlmaLinux host as we've installed Grafana and Loki. That's practical because AxoSyslog can receive data from a wide range of sources and protocols, including syslog and OpenTelemetry. This also means that if you upgrade or replace your existing collectors or log relays with AxoSyslog, you can forward your logs between the nodes using the OpenTelemetry Protocol (OTLP), which is a quick and easy way to start modernizing your logging pipeline. That way your data travels reliably via gRPC, all the way to Loki. (Note to self: we should really do a blog on the awesomeness of gRPC/OTLP for log transport.)

Of course, you can install AxoSyslog on a different host, and send data from that host to Loki. You can also install AxoSyslog on other distributions, or using Docker/Podman/Helm.

First, enable the EPEL repositories, which contain some of AxoSyslog's dependencies:

sudo dnf install epel-release
sudo dnf config-manager --set-enabled crb

Add the AxoSyslog repository:

sudo tee /etc/yum.repos.d/axosyslog.repo <<< '[axosyslog]
name=AxoSyslog
baseurl=https://pkg.axoflow.io/rpm/stable/almalinux-9/$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://pkg.axoflow.io/axoflow-code-signing-pub.asc' > /dev/null

Update the packages list:

sudo dnf update -y

Then install AxoSyslog:

sudo dnf install "axosyslog-*" --exclude=axosyslog-logrotate

(If you don't want to install every sub-package install at least the axosyslog and the axosyslog-grpc packages (that's axosyslog-mod-grpc for DEB based distributions). AxoSyslog doesn't support sending data to Loki via HTTP.)

Configure AxoSyslog

Edit the AxoSyslog configuration file /etc/syslog-ng/syslog-ng.conf.

Configure a Loki destination. Here we just add a few simple labels about the sender host and application, but you can add other syslog-ng macros as needed, even values parsed dynamically from the content of the log message. (AxoSyslog has a powerful data processing engine called Filterx that can handle complex data structures.)

destination d_loki {
  loki(
    url("localhost:9096")
    labels(
        "app" => "$PROGRAM",
        "host" => "$HOST",
        "test_label" => "static",
    )
    workers(1)
    batch-timeout(2000)
    batch-lines(20)
  );
};

The default AxoSyslog configuration doesn't accept logs via network connections, so to make testing easy, add a source for common network sources:

source s_net { default-network-drivers(); };

Create a log path that connects the source to the destination:

log { source(s_net); destination(d_loki); };

Depending on which ports you want to receive logs with AxoSyslog, you might have to adjust your SELinux policies (or for a quick test, just put SELinux into permissive mode):

sudo setenforce 0

Check that everything is fine with the configuration, then restart AxoSyslog:

sudo syslog-ng --syntax-only
sudo systemctl restart syslog-ng

You might get an error message like this for the Loki destination: Error parsing config: syntax error, unexpected LL_IDENTIFIER, expecting '}' This means that you (like me) forgot to install the axosyslog-grpc package, so run sudo dnf install axosyslog-grpc

After that, send some test logs using loggen:

loggen --stream --size 300 --rate 100 --interval 10 127.0.0.1 514

Check logs in Grafana

Let’s check the logs that we sent to Loki. Open Grafana in your browser, then select Explore.

‍

In the Label filters field select host, and select localhost (or the host your AxoSyslog deployment sent messages from) for the label value, then click Run query. The logs you sent show up:

Now, obviously, the well-padded logs of loggen are not too interesting, so let's send some real-life logs from a FortiGate firewall:

while true; do echo "<165> fortigate date=$(date -u +%Y-%m-%d) time=$(date -u +"%H:%M:%S%Z") devname=us-east-1-dc1-a-dmz-fw devid=FGT60D4614044725 logid=0100040704 type=event subtype=system level=notice vd=root logdesc=\"System performance statistics\" action=\"perf-stats\" cpu=2 mem=35 totalsession=61 disk=2 bandwidth=158/138 setuprate=2 disklograte=0 fazlograte=0 msg=\"Performance statistics: average CPU: 2, memory:  35, concurrent sessions:  61, setup-rate: 2\""; sleep 1; done | nc -v 127.0.0.1 514

Since this log message is not a well-formed syslog message (more on this problem in the Fix the syslog mess blog), these will show up on for the host=127.0.0.1 query, or just use app="fortigate":

With that, you can start building dashboards in Grafana, or revisit the AxoSyslog configuration to add more labels (and maybe give a try to our FilterX data processing solution in AxoSyslog):

Conclusion

In this post we’ve shown how easy it is to send syslog data to Grafana Loki using AxoSyslog, our binary-compatible syslog-ng™ replacement. In addition to sending your logs directly to Loki, AxoSyslog provides dynamic metadata labeling to make your data easier to search and more accessible for Grafana dashboards. Give it a try, and let us know what you think! In case you run into an issue or need help getting started, feel free to contact us.