AxoSyslog - Log Collection for Kubernetes

Recently, we have released our custom cloud-ready images of syslog-ng. Continuing the cloud native integration of syslog-ng this article explains how to apply these images to solve an important Kubernetes use-case: the collection of Kubernetes metadata.

syslog-ng has a long history for being a trusted tool that provides unprecedented performance, stability and flexibility, both when deployed in a collector or an aggregator role. This Kubernetes integration bridges the gap between cloud native and enterprise deployments, especially when they operate side-by-side.

This post shows you how to install this syslog-ng image using the AxoSyslog Helm charts and use it to send Kubernetes logs into OpenSearch, as a means to demonstrate how it all works.

If you are planning to try AxoSyslog in production we recommend doing that via Logging operator that supports syslog-ng as a aggregator agent since version 4.0.

syslog-ng kubernetes() source

In the syslog-ng configuration, the entities where syslog-ng reads or receives logs from are called sources. The kubernetes() source collects logs from the Kubernetes cluster where syslog-ng is running. It follows the design of CRI logging, reading container logs from /var/log/containers or /var/log/pods.

Thanks to syslog-ng’s versatile python binding support, after reading the logs from the container log files, the kubernetes() source enriches them with various metadata, acquired via the kubernetes python client.

Syslog-ng 4.1.1 retrieves the following metadata fields:

You can find more details on how these are calculated in the source codes of syslog-ng’s kubernetes SCL, KubernetesAPIEnrichment, and kubernetes-client’s CoreV1Api. We are working on extending this list with even more fields, like namespace labels, so stay tuned!

axosyslog-collector Helm chart

Helm is the package manager of Kubernetes. It helps to define, install, and upgrade even the most complex Kubernetes applications. Axoflow’s axosyslog-charts repository houses the axosyslog-collector Helm chart, which we will use for demonstrating the kubernetes() source. The chart uses the Alpine-based syslog-ng docker image built by Axoflow. The axosyslog-collector is a rudimentary chart for collecting Kubernetes logs, it can:

• forward logs conforming to RFC3164 via TCP/UDP connections, or
• send them to OpenSearch.

If you need a complete Kubernetes logging solution, we recommend checking the Logging Operator.

Demo: Sending Kubernetes logs to OpenSearch

The following tutorial shows you how to send Kubernetes logs to OpenSearch.

Prerequisites

You will need a Kubernetes cluster. We used minikube with docker driver and Helm. We used a Ubuntu 22.04 (amd64) machine, but it should work on any system that can run minikube (2 CPUs, 2GB of free memory, 20GB of free disk space).

The OpenSearch service needs a relatively large mmap count setting, so set it to at least 262144, for example:

Generate logs

Install kube-logging/log-generator to generate logs.‍

Check that it’s running:

The output should look like:‍

Set up OpenSearch

Install an OpenSearch cluster with Helm:

Now you should have 5 pods. Check that they exist:

The output should look like:

Forward the 5601 port of the OpenSearch Dashboards service (replace the name of the pod with your pod).

The output should look like:

You can login to the dashboard at http://localhost:8080 with admin/admin. You will soon create an Index Pattern here, but first you have to send some logs from syslog-ng.

Set up axosyslog-collector

First, add the AxoSyslog Helm repository:

Create a YAML file (axoflow-demo.yaml) to configure the collector.

Check what the syslog-ng.conf file will look like with your custom values:

The output should look like:

Install the axosyslog-collector chart:

The output should look like:

Check your pods:

The output should look like:

Check the logs in OpenSearch

Head back to OpenSearch Dashboards, and create an Index Pattern called “test-axoflow-index”: http://localhost:8080/app/management/opensearch-dashboards/indexPatterns. At Step 2, set the Time field to “@timestamp”.

Now you can see your logs on the Discover view http://localhost:8080/app/discover. Opening the detailed view for a log entry shows you the fields sent to OpenSearch.

Summary

From this post you have learned:

• How to install a syslog-ng log collector using our AxoSyslog charts, which is one of the first elements of our new open source project, the AxoSyslog cloud native syslog-ng distribution.
• Configure it to send your logs into OpenSearch.
• Learned how AxoSyslog enriches your log messages with Kubernetes metadata.