This post shows you how you can monitor the log traffic of your Kubernetes deployments if you are using the AxoSyslog log collector to collect your logs. AxoSyslog provides detailed metrics to bring observability to your logging and observability data infrastructure.

In the previous blog post (AxoSyslog – Log Collection for Kubernetes), we showed you how you can install the AxoSyslog Collector using Axoflow’s Helm charts, and enrich your log messages with Kubernetes metadata.

AxoSyslog is a cloud-native distribution of syslog-ng, which has a long history of being a trusted tool that provides unprecedented performance, stability, and flexibility, both when deployed in a collector or an aggregator role. Axoflow’s Kubernetes integration bridges the gap between cloud-native and enterprise deployments, especially when they operate side-by-side.

This blog post is a continuation of the above post, in which we will show you how you can monitor and visualize the metrics of your AxoSyslog Kubernetes deployments. Everything you’ll see in this blog builds on top of syslog-ng’s revamped Prometheus-based metric system. That and the whole AxoSyslog distribution aims to bring syslog-ng closer to the cloud-native ecosystem, focusing on extensive observability.

If you are planning to try AxoSyslog in production we recommend doing that via Logging operator that supports syslog-ng as an aggregator agent since version 4.0.

Before you begin

You will need: 

• a Kubernetes cluster (Minikube, kind, or K3s can help you get started easily),
• Helm, and
• a recent kubectl version.

About the demo chart

First, you are going to install a demo umbrella Helm chart prepared for this blog post to skip some time-consuming inconvenience: https://github.com/MrAnno/axoblog

The chart pulls and integrates all the essential components together needed for the AxoSyslog metric visualization:

• the AxoSyslog Collector
• a Prometheus server
• a demo syslog-ng metric exporter
• a prepared Grafana dashboard

Install the Demo Helm Chart

To install the demo chart, clone the following repository and download/build the chart dependencies:

git clone https://github.com/MrAnno/axoblog.git
cd axoblog/axosyslog-kube-metrics
helm dependency build

Configure the AxoSyslog collector to forward Kubernetes logs to OpenSearch by creating demo-values.yaml with the following content:

axosyslog-collector:
  config:
    sources:
      kubernetes:
        enabled: true
    destinations:
      opensearch:
      - address: "opensearch-cluster-master"
        index: "test-index"
        user: "admin"
        password: "admin"
        tls:
          peerVerify: false
        template: "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))"

Install OpenSearch and log-generator so that you can produce a basic log flow for the sake of the demo:

sysctl -w vm.max_map_count=262144

helm repo add opensearch https://opensearch-project.github.io/helm-charts/
helm repo add kube-logging https://kube-logging.github.io/helm-charts/

helm repo update

helm install demo-opensearch --wait opensearch/opensearch
helm install demo-log-generator --wait kube-logging/log-generator

Install the demo axosyslog-kube-metrics umbrella chart:

cd ..
helm install metrics-demo --wait -f axosyslog-kube-metrics/demo-values.yaml \
    ./axosyslog-kube-metrics/ --post-renderer ./axosyslog-kube-metrics/kustomize.sh

Follow the on-screen instructions to acquire an admin password and to perform a port-forward on the Grafana deployment.

Wait a few minutes until the system collects some logs.

Log ingress/egress metrics

Log into Grafana at http://localhost:3000 with the admin user, and find the prepared AxoSyslog dashboard. You should see something like this:

The first visualization shows the rate of all events produced by your Kubernetes cluster per Pod. This is useful for real-time monitoring of your deployments’ logging activity.

The “Log Ratio of Pods” pie chart shows the current ratio of these events by Pods. The fourth bar chart does something similar, it shows per-Pod ratios over time, while also showing the actual number of logs sent by the pods.

“Output Events” is for log egress visualization. In addition to displaying delivered events, it can also indicate if AxoSyslog queues are filling up due to overload, or if the log destination is not available and you are losing messages.

Generate additional logs

You can generate additional logs to test the dashboard using the log generator, for example:

export POD_NAME=$(kubectl get pods -l "app.kubernetes.io/name=log-generator,app.kubernetes.io/instance=demo-log-generator" -o jsonpath="{.items[0].metadata.name}")

kubectl port-forward $POD_NAME 11000

curl -X POST 'http://localhost:11000/loggen' -H 'Content-Type: application/json' \
  --data-raw '{ "type": "syslog", "format": "kube.cri", "count": 100 }'

curl -X POST 'http://localhost:11000/loggen' -H 'Content-Type: application/json' \
  --data-raw '{ "type": "syslog", "format": "kube.json", "count": 1000 }'

AxoSyslog metrics overview

By editing the preconfigured visualizations, you can check the metrics they are based on.

AxoSyslog provides a wide variety of metrics by default with detailed labeling.

The most significant metrics are the following:

Metrics are also available in bytes, they can come in handy when you are trying to calculate billing estimates (for example,  output_event_bytes_total).

You can also create your own metrics using the metrics-probe() parser in AxoSyslog. By default, it produces basic classification on logs, which is available under the syslogng_classified_events_total key.

When creating your own metrics in AxoSyslog, always make sure that the cardinality of your metrics can not grow infinitely large within a reasonable time interval.

Optimizing for observability with the power of open source

While we at Axoflow are building our own observability supply chain, we truly believe in the open-source culture, that’s why we strive to contribute the heart (the building blocks) of our products back to open source.

We did the same this time as well, and contributed the following to the syslog-ng project:

• all new metrics mentioned in this post, as well as 
• the metrics-probe() parser, which allows you to create comprehensive metrics with deeper granularity. 

We actively use them, for example, in AxoRouter for classification purposes: