Install AxoSyslog

• 1: Install AxoSyslog with Podman and systemd
• 2: Install AxoSyslog with Helm
• 2.1: Parameters of the AxoSyslog collector Helm chart
• 3: Install AxoSyslog with Podman
• 4: Install AxoSyslog with Docker

1 - Install AxoSyslog with Podman and systemd

This page shows you how to run AxoSyslog as a systemd service using podman.

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

• They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
• They incorporate cloud-native features and settings, such as the Kubernetes source.
• They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
• They support the ARM architecture.

The AxoSyslog images support the following architectures:

• amd64
• arm/v7
• arm64

Prerequisites

Podman version 4.6.1.

The steps in this procedure were tested on CentOS 9, but should work on other similar distributions as well.

Install AxoSyslog as a systemd service

1. Make sure that there is no axosyslog.service unit file on the system. Run the following commands:

   sudo rm /etc/systemd/system/axosyslog.service
   

   Expected output:

   rm: cannot remove '/etc/systemd/system/axosyslog.service': No such file or directory
   

   sudo systemctl cat axosyslog.service
   

   Expected output:

   No files found for axosyslog.service.
   

2. Create a systemd unit file called /etc/containers/systemd/axosyslog.container based on the following template:

   sudo curl -o /etc/containers/systemd/axosyslog.container https://axoflow.com/docs/axosyslog-core/install/podman-systemd/axosyslog.container
   

3. Edit the unit file as needed for your environment.

   We recommend using the default mount points:

   Purpose	On the host	In the container
   Disk-buffer and persist files	/var/lib/syslog-ng	/var/lib/syslog-ng
   syslog-ng configuration file	/opt/axosyslog/etc	/etc/syslog-ng
   Output log files	/opt/axosyslog/var/log	/var/log

4. (Optional) Create an override.conf file to set custom environment values. This can be useful if you don’t want to modify /etc/containers/systemd/axosyslog.container. Run:

   systemctl edit axosyslog
   

   Later you can edit this file by running the previous command again.

5. Create the /opt/axosyslog/etc/syslog-ng.conf configuration file based on the following template.

   sudo mkdir -p /opt/axosyslog/etc/ ; sudo curl -o /opt/axosyslog/etc/syslog-ng.conf https://axoflow.com/docs/axosyslog-core/install/podman-systemd/syslog-ng.conf
   

   With the following sample configuration file AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages file.

   You can customize the configuration file according to your needs. For a few pointers, see Configuring AxoSyslog on server hosts and the rest of this guide.

6. Run the following commands to reload the systemd configuration and launch the axosyslog service. Though the systemctl commands are run as root, the container will run as the specified user if set appropriately in the unit file.

   sudo systemctl daemon-reload
   sudo systemctl stop axosyslog
   sudo systemctl start axosyslog
   

   If there aren’t any errors, these commands don’t have any output.

7. Run the following command to verify that the service was properly started:

   journalctl -b -u axosyslog | tail -100
   

   The output should be similar to:

   Feb 12 09:04:40 <your-hostname> systemd[1]: Starting AxoSyslog Container...
   Feb 12 09:04:40 <your-hostname> podman[2783]: 2024-02-12 09:04:40.454665314 -0500 EST m=+0.167732500 system refresh
   Feb 12 09:04:40 <your-hostname> axosyslog[2783]: Trying to pull ghcr.io/axoflow/axosyslog:latest...
   Feb 12 09:04:40 <your-hostname> axosyslog[2783]: Pulling image //ghcr.io/axoflow/axosyslog:latest inside systemd: setting pull timeout to 5m0s
   Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Getting image source signatures
   Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
   Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:619be1103602d98e1963557998c954c892b3872986c27365e9f651f5bc27cab8
   Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:b061f41886afb563aff2a5f731f3286ba54ea6f657ed3e282f5339a12a64c5ef
   Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:1b8d965a650c6a05227bd5c549930c9898071e8e7abb26886d4169a99762de0a
   Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:b5b0ce6ebef193c4f909379188cfb59443e8a1809816fbb476074908b170b4d1
   Feb 12 09:04:50 <your-hostname> axosyslog[2783]: Copying config sha256:c379d94ef2c5ec348dfb3a93eed9a19aed667c396008db85edc354c8f4f8cb6a
   Feb 12 09:04:50 <your-hostname> axosyslog[2783]: Writing manifest to image destination
   Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.422390687 -0500 EST m=+10.135457863 container create 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.url=https://axoflow.io/, org.opencontainers.image.source=https://github.com/axoflow/axosyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.title=AxoSyslog, org.opencontainers.image.vendor=Axoflow, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, maintainer=axoflow.io, org.opencontainers.image.licenses=GPL-3.0-only)
   Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.402626446 -0500 EST m=+10.115693622 image pull c379d94ef2c5ec348dfb3a93eed9a19aed667c396008db85edc354c8f4f8cb6a ghcr.io/axoflow/axosyslog:latest
   Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.489925509 -0500 EST m=+10.202992695 container init 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.licenses=GPL-3.0-only, org.opencontainers.image.vendor=Axoflow, maintainer=axoflow.io, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.url=https://axoflow.io/, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.title=AxoSyslog, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, org.opencontainers.image.source=https://github.com/axoflow/axosyslog)
   Feb 12 09:04:50 <your-hostname> systemd[1]: Started AxoSyslog Container.
   Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.500050669 -0500 EST m=+10.213117845 container start 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.source=https://github.com/axoflow/axosyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.licenses=GPL-3.0-only, org.opencontainers.image.vendor=Axoflow, org.opencontainers.image.title=AxoSyslog, maintainer=axoflow.io, org.opencontainers.image.url=https://axoflow.io/)
   Feb 12 09:04:50 <your-hostname> axosyslog[2783]: 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f
   Feb 12 09:04:50 <your-hostname> AxoSyslog[2821]: [2024-02-12T14:04:50.806054] syslog-ng starting up; version='4.6.0'
   

8. Send a test message to the service:

   echo '<5> localhost test: this is a test message' | nc localhost 514
   

   Check that the test message has arrived into the log file:

   less /opt/axosyslog/var/log/messages
   

   The output should be similar to:

   Feb 19 15:49:12 localhost test: this is a test message
   

Customize the configuration

To customize the configuration, edit the /opt/axosyslog/etc/syslog-ng.conf file on the host, then reload the service.

Managing the AxoSyslog systemd service

• You can reload syslog-ng running in the container via systemctl. The following command reloads the syslog-ng.conf file, without stopping/starting syslog-ng itself.

  sudo systemctl reload axosyslog
  

• You can access syslog-ng-ctl from the host, for example by running:

  podman exec -ti AxoSyslog syslog-ng-ctl show-license-info
  

  If you use syslog-ng-ctl regularly, you can create the /opt/axosyslog/bin/syslog-ng-ctl file with the following content, make it executable, and add it to your path. That way running syslog-ng-ctl <command> will execute the command in the AxoSyslog container.

  #!/bin/bash  
  
  podman exec -ti AxoSyslog syslog-ng-ctl "$@"  
  

• The traditional method of starting a service at boot (systemctl enable) is not supported for container services. To automatically start the AxoSyslog service, make sure that the following line is included in the unit file. (It is included in the sample template.)

  [Install]
  WantedBy=default.target
  

2 - Install AxoSyslog with Helm

AxoSyslog provides Helm charts for syslog-ng. You can use these charts to install cloud-ready syslog-ng images created and maintained by Axoflow.

Prerequisites

You must have Helm 3.0 or newer installed to use these charts. Refer to the official Helm documentation for details.

Limitations

The chart provides parameters that make it easy to:

• collect logs using the kubernetes() source, and
• forward the logs using the network() and opensearch() destinations.

To use other sources and destinations, use the config.raw parameter. For the list of configurable parameters and their default values, see Parameters of the AxoSyslog collector Helm chart.

Install

To install the axosyslog-collector charts, complete the following steps.

1. Clone the chart repository.

   helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
   helm repo update
   

2. Install the chart. The following command installs axosyslog-collector into the default namespace. For the list of configurable parameters and their default values, see Parameters of the AxoSyslog collector Helm chart. If you want to use disk-buffers, see also How to use disk-buffers in containers and Kubernetes.

   helm install --generate-name axosyslog/axosyslog-collector
   

   NAME: axosyslog-collector-1683469360
   LAST DEPLOYED: Sun May  7 16:22:40 2023
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   1. Watch the axosyslog-collector-1683469360 container start.
     $ kubectl get pods --namespace=default -l app=axosyslog-collector-1683469360 -w
   

3. Check that the pod is running.

   kubectl get pods
   

   The output should look like:

   NAME                                   READY   STATUS    RESTARTS   AGE
   axosyslog-collector-1683469360-tptfb   1/1     Running   0          28s
   

How to use disk-buffers in containers and Kubernetes

When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.

• Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
  • In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
  • In a container, mount the disk-buffer directory from the host, or store it on a local volume.
• Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
• Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.

If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.

Uninstall

Tip: List all installed releases using helm list.

To uninstall a chart release, run:

helm delete <name-of-the-release-to-delete>

2.1 - Parameters of the AxoSyslog collector Helm chart

The following table lists the configurable parameters of the AxoSyslog collector chart and their default values. For details on installing the chart, see Install AxoSyslog with Helm.

Parameters for syslog-ng configuration

The following example uses the config.raw parameter to configure a custom destination:

config:
  raw: |
    @version: 4.5.0
    @include "scl.conf"

    log {
      source {
        syslog(port(12345));
      };

      destination {
        logscale(
          token("your-secret-humio-ingest-token")
        );
      };

      flags(flow-control);
    };

daemonset:
  hostNetworking: true

Network destination

Send logs over the network, conforming to RFC3164 using the network() destination.

For example:

config:
  destinations:
    network:
      - transport: tcp
        address: localhost
        port: 12345
        template: "$(format-json .*)"

OpenSearch destination

Send logs to OpenSearch over HTTP or HTTPS.

For example:

config:
  destinations:
    opensearch:
      - address: 10.104.232.94
        index: "test-axoflow-index"
        tls:
          CAFile: "/path/to/CAFile.pem"
          CADir: "/path/to/CADir/"
          Cert: "/path/to/Cert.pem"
          Key: "/path/to/Key.pem"
          peerVerify: true
          template: "$(format-json .*)"

Generic parameters

3 - Install AxoSyslog with Podman

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

• They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
• They incorporate cloud-native features and settings, such as the Kubernetes source.
• They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
• They support the ARM architecture.

The AxoSyslog images support the following architectures:

• amd64
• arm/v7
• arm64

Install the AxoSyslog images

You can find the list of tagged versions at https://github.com/axoflow/axosyslog-docker/pkgs/container/axosyslog.

To install the latest stable version, run:

podman pull ghcr.io/axoflow/axosyslog:latest

You can also use it as a base image in your Dockerfile:

FROM ghcr.io/axoflow/axosyslog:latest

If you want to test a development version, you can use the nightly builds:

podman pull ghcr.io/axoflow/axosyslog:nightly

Note: These named packages are automatically updated when a new package is released. To install a specific version, run podman pull ghcr.io/axoflow/axosyslog:<version-number>, for example:

podman pull ghcr.io/axoflow/axosyslog:4.5.0

Customize the configuration

The AxoSyslog container image stores the configuration file at /etc/syslog-ng/syslog-ng.conf. By default, AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages and /var/log/messages-kv.log files using this configuration file from the syslog-ng repository.

To customize the configuration, create your own configuration file and override the file in the container image with it, for example:

podman run --rm --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

How to use disk-buffers in containers and Kubernetes

When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.

• Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
  • In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
  • In a container, mount the disk-buffer directory from the host, or store it on a local volume.
• Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
• Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.

If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.

Expose port to receive incoming traffic

To receive incoming network in a container, you must expose the port from the container where you want to receive the traffic to the host that’s running the container. Typically, this is only needed if you are running AxoSyslog as a relay or a server/aggregator.

By default, the AxoSyslog container images expose the ports commonly used to receive syslog traffic:

• 514/udp, typically used for RFC3164 (BSD-syslog) formatted traffic.
• 601/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic.
• 6514/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic over TLS.

To expose a specific port, use the --expose option when starting the container. Make sure to include the IP address of the host to make the port externally accessible.

For example, if you are receiving OpenTelemetry messages using the opentelemetry() source, expose the 4317 port:

podman run --rm --expose 127.0.0.1:4317:4317/tcp --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

4 - Install AxoSyslog with Docker

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

• They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
• They incorporate cloud-native features and settings, such as the Kubernetes source.
• They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
• They support the ARM architecture.

The AxoSyslog images support the following architectures:

• amd64
• arm/v7
• arm64

Install the AxoSyslog images

You can find the list of tagged versions at https://github.com/axoflow/axosyslog-docker/pkgs/container/axosyslog.

To install the latest stable version, run:

docker pull ghcr.io/axoflow/axosyslog:latest

You can also use it as a base image in your Dockerfile:

FROM ghcr.io/axoflow/axosyslog:latest

If you want to test a development version, you can use the nightly builds:

docker pull ghcr.io/axoflow/axosyslog:nightly

Note: These named packages are automatically updated when a new package is released. To install a specific version, run docker pull ghcr.io/axoflow/axosyslog:<version-number>, for example:

docker pull ghcr.io/axoflow/axosyslog:4.5.0

Customize the configuration

The AxoSyslog container image stores the configuration file at /etc/syslog-ng/syslog-ng.conf. By default, AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages and /var/log/messages-kv.log files using this configuration file from the syslog-ng repository.

To customize the configuration, create your own configuration file and override the file in the container image with it, for example:

docker run --rm --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

How to use disk-buffers in containers and Kubernetes

When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.

• Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
  • In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
  • In a container, mount the disk-buffer directory from the host, or store it on a local volume.
• Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
• Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.

If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.

Expose port to receive incoming traffic

To receive incoming network in a container, you must expose the port from the container where you want to receive the traffic to the host that’s running the container. Typically, this is only needed if you are running AxoSyslog as a relay or a server/aggregator.

By default, the AxoSyslog container images expose the ports commonly used to receive syslog traffic:

• 514/udp, typically used for RFC3164 (BSD-syslog) formatted traffic.
• 601/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic.
• 6514/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic over TLS.

To expose a specific port, use the --expose option when starting the container. Make sure to include the IP address of the host to make the port externally accessible.

For example, if you are receiving OpenTelemetry messages using the opentelemetry() source, expose the 4317 port:

docker run --rm --expose 127.0.0.1:4317:4317/tcp --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest