Install AxoSyslog with Helm

AxoSyslog provides a Helm chart. You can use this chart to install cloud-ready syslog-ng images created and maintained by Axoflow.

Prerequisites for the Helm chart

To use this chart, you need:

• A Kubernetes cluster that runs Kubernetes 1.22 or newer.
• kubectl, configured to access your cluster.
• Helm 3.0 or newer. For details, see the official Helm documentation.

Collector and aggregator use cases

The chart has parameters for the following use cases:

• As a collector: The collector uses the kubernetes() source to collect the local logs. It sends the logs to the aggregator, to a syslog server, to an OpenSearch node, or to a different AxoSyslog node.
• As an aggregator: The aggregator receives RFC3164 and RFC5424 syslog messages from all senders, and axosyslog-otlp() messages from other AxoSyslog nodes. It stores the messages locally, or sends them to remote destinations.

By default, the collector sends the logs to the aggregator. You can configure and deploy the two components independently. To use other sources and destinations, use the config.raw parameter of the collector or the aggregator. For the list of parameters and their default values, see Parameters of the AxoSyslog Helm chart.

Install the Helm chart

To install the axosyslog chart, complete the following steps.

1. Add the chart repository.

   helm repo add axosyslog https://axoflow.github.io/axosyslog
   helm repo update

2. Install the chart. The default settings install the following into the default namespace:

   • A collector DaemonSet. It collects the pod logs on each node, and sends them to the aggregator.
   • An aggregator StatefulSet. It receives syslog and axosyslog-otlp() messages, and writes them to a file.

   To install only one component, disable the other component: set collector.enabled=false or aggregator.enabled=false. For the list of parameters and their default values, see Parameters of the AxoSyslog Helm chart. If you use disk-buffers, also read How to use disk-buffers in containers and Kubernetes.

   • Install with the default values:

     helm install --generate-name axosyslog/axosyslog

   • Install only the collector:

     helm install --generate-name axosyslog/axosyslog --set aggregator.enabled=false

   • Install only the aggregator:

     helm install --generate-name axosyslog/axosyslog --set collector.enabled=false

   The output should be similar to:

   NAME: axosyslog-1713953907
   LAST DEPLOYED: Wed Apr 24 12:18:28 2024
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   1. Watch the axosyslog-1713953907 containers start.
     $ kubectl get pods --namespace=default -l app.kubernetes.io/instance=axosyslog-1713953907,app.kubernetes.io/name=axosyslog -w

   The NAME field shows the name of the release. In the following commands, replace <release-name> with this name.

3. Check that the pods are running.

   kubectl get pods

   The output should list the pods that are running: a collector pod on every node and an aggregator pod for the default settings. For example, on a single-node cluster:

   NAME                                   READY   STATUS    RESTARTS   AGE
   axosyslog-1713953907-collector-ddftq   1/1     Running   0          57s
   axosyslog-1713953907-aggregator-0      1/1     Running   0          57s

4. Configure the settings of the pods for your use case.

   1. Create a file called my-values.yaml.

   2. Add the configuration needed for your use case. The settings in this file override the default configuration settings of the chart.

   3. Update your deployment using the my-values.yaml file by running:

      helm upgrade <release-name> axosyslog/axosyslog -f my-values.yaml

      The output should be similar to:

      Release "axosyslog-1713953907" has been upgraded. Happy Helming!
      ...

      Tip To list the non-default values of a release, run helm get values <release-name>.

Send the collector logs to another destination

By default, the collector sends the logs in JSON format to the aggregator over TCP. To send the logs to a different destination, configure the destination in your values file, then run helm upgrade. For example, the following values file sends the logs in JSON format to the 192.0.2.10:514 address over TCP:

collector:
  config:
    destinations:
      syslog:
        enabled: true
        transport: tcp
        address: 192.0.2.10
        port: 514
        template: "$(format-json .*)"

For details and other parameters, see Collector parameters.

Send test messages to the aggregator

To make sure that the aggregator receives messages, send test messages from the aggregator pod.

1. Run loggen in the aggregator pod:

   kubectl exec <release-name>-aggregator-0 -- loggen -S 127.0.0.1 1514

   Expected output:

   count=9328, rate = 882.83 msg/sec
   count=9786, rate = 884.20 msg/sec
   count=9800, rate = 27.92 msg/sec
   average rate = 928.58 msg/sec, count=9800, time=10.5538, (average) msg size=256, bandwidth=232.14 kB/sec

2. Check the configured destinations for the generated messages. For example, with the default file destination:

   kubectl exec <release-name>-aggregator-0 -- tail -n 5 /var/log/syslog

   The generated messages look like this:

   2024-05-02T10:56:31.000000+00:00 localhost prg00000[1234]: seq: 0000000065, thread: 0000, runid: 1714647391, stamp: 2024-05-02T10:56:31 PADDPADDPADDPADD

How to use disk-buffers in containers and Kubernetes

When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.

• Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
  • In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
  • In a container, mount the disk-buffer directory from the host, or store it on a local volume.
• Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
• Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.

If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.

Upgrade the Helm chart

To upgrade the chart to a newer version, and to roll back an upgrade, see Upgrade the Helm chart.

Uninstall the Helm chart

To uninstall a release of the chart, run:

helm uninstall <release-name>