From syslog-ng to LogScale: structured logs from any source

Syslog-ng is a well-known and widely used, enterprise-grade solution for collecting, transforming, and transporting logging and observability data. Falcon LogScale is an enterprise tool that provides fast search, analysis, and visualization of all of your data. This post shows you how to send structured log messages from syslog-ng to your Falcon LogScale deployment using its Ingest Structured Data API. That way you don’t have to parse the data on Falcon LogScale, because syslog-ng already sends it in a structured format that LogScale understands.

We use AxoSyslog, the cloud-native distribution of syslog-ng by Axoflow in our examples, but you can use practically the same configuration with any syslog-ng instance version 4.3.1 or later. The overall architecture is really simple, and looks like this:

Why use syslog-ng instead of the Falcon LogScale Collector?

Unlike the limited scope of vendor-supplied collectors, syslog-ng allows you to collect logs from an extremely wide variety of sources – far more than the original design from 20 years ago. It can receive logs and metrics from over twenty different sources, ranging from legacy (like SNMP) to modern (like OpenTelemetry) technologies, and all at far greater scale than proprietary syslog listeners. It indeed can serve as a universal collector in many cases.

Often the different departments and groups of your organization use different tools to ingest and process the logs and metrics from your business, because they use them for different purposes. For example, your security department may use a SIEM to keep your business secure, while your IT and DevOps departments use another tool to help them keep your systems running.

With syslog-ng, you can send your logs to multiple destinations (not only to LogScale), so you don’t have to deploy multiple logging agents on each host. With syslog-ng’s powerful filtering and log routing capabilities, you don’t have to send the same set of logs to every destination, but can select only the relevant ones for each department. This reduces the noise, and also decreases the volume of logs that any particular department has to process and store, thus lowering the costs.

Prerequisites

Before you begin, you will need:

an Ingest token for your LogScale repository where you want to send the logs,
a Kubernetes cluster (Minikube, kind, or K3s can help you get started easily),
Helm, and
a recent kubectl version.

Create AxoSyslog configuration

First, you need a configuration file for AxoSyslog. Copy the following configuration into a file called logscale.yaml. (If you are configuring syslog-ng directly on a host using syslog-ng.conf, you need only the value of the raw key (without the leading |).)

config:
  raw: |
    @version: 4.3
    @include "scl.conf"

    log {
      source {
        syslog(port(12345));
      };

      destination {
        logscale(
          token("your-secret-humio-ingest-token")
        );
      };

      flags(flow-control);
    };

daemonset:
  hostNetworking: true

If needed, edit the configuration file. For example:

By default, the logscale() destination of syslog-ng sends the messages to the “https://cloud.humio.com” endpoint. If you want to use a different one, for example, the endpoint of the Community Edition, use the url() option:
```
logscale(
          url("https://cloud.community.humio.com")
          token("your-secret-humio-ingest-token")
        );
```
By default, the logscale() destination sends the RFC5424 fields as attributes. If you want to send different fields (such as those from Splunk Connect for Syslog (SC4S)), you can override the default template via the attributes() option. If your incoming data does not conform to the RFC5424 standard, other fields (such as those created by other syslog-ng parsers) can be used with a template supplied via the attributes() option as well. Stay tuned for a followup to this blog that discusses expanded filtering and curation capabilities of Axoflow routing components, which will be based on syslog-ng as well.

Install AxoSyslog Collector

Install the AxoSyslog collector on your cluster with the configuration file you created in the previous step.

helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
helm repo update
helm install axosyslog-logscale-demo axosyslog/axosyslog-collector -f logscale.yaml

Send log messages

Now you can generate some test messages and send them to the AxoSyslog collector. The sample command sends messages in RFC5424 syslog format, which is compatible with the default attributes() template. The messages include SDATA fields which will be ingested and rendered by LogScale in a structured way. Run the following command.

kubectl exec $(kubectl get pod -l app=axosyslog-logscale-demo-axosyslog-collector -o jsonpath="{.items[0].metadata.name}") -- loggen -i -S -P -p '[meta sequenceId="5" sysUpTime="6"][origin ip="zts-001" software="syslog-ng"]' -r 10 localhost 12345

AxoSyslog receives the messages and forwards them to LogScale.

LogScale UI

Open the LogScale UI and navigate to your repository. You can see that LogScale shows the RFC5424 fields in a structured manner as separate columns.

Summary

Built on syslog-ng, AxoSyslog is a versatile and robust tool that can send your data to Falcon LogScale in a structured format that LogScale understands, so it doesn’t need to parse the data. Using AxoSyslog improves the reliability and performance of your observability data pipeline, and at the same time it simplifies your infrastructure, because you won’t need other log collector agents on the same host.

For an overview on how our platform enhances syslog-ng based log collection with metrics, see the Metrics for syslog-ng based log management infrastructures blog post.