This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

AxoSyslog documentation

AxoSyslog is a cloud-native syslog-ng distribution by Axoflow. If you want to try running syslog-ng in a container, or use it as a log collector in Kubernetes, try AxoSyslog!

This documentation is originally based on the syslog-ng Open Source Edition documentation for version 3.38 and is used in compliance with the terms of The syslog-ng Open Source Edition Documentation License. We are continuously working on it to keep it up-to-date with the latest released version of the syslog-ng project (currently 4.5).

In compliance with the above license, syslog-ng is used only to refer to the binary of the application, or in filenames, pathnames, and similar technical terms. For all other references, we use the AxoSyslog name.

New files and modifications compared to the syslog-ng Open Source Edition documentation are under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks.

The Axoflow and AxoSyslog trademarks and logos are trademarks of Axoflow Inc. All other trademarks are property of their respective owners.

This documentation is maintained by Axoflow, and continually updated for new releases.

Acknowledgements

We would like to express our gratitude to everyone involved with the project, including the syslog-ng developers, advocates, and documentation maintainers who worked at BalaBit or are still working at One Identity, or who helped the project as external contributors, users, or as part of the community.

Feedback

Any feedback is greatly appreciated, especially on what else this document should cover. General comments, errors found in the text, and any suggestions about how to improve the documentation are also welcome as GitHub issues, or on the Axoflow Discord server.

Support

In case you need help with any of the AxoSyslog projects, or directly with syslog-ng, you have several ways to contact us:

We also provide consulting and professional services for logging and observability related projects. Contact us if you need our help!

1 - What's new

This page is a changelog that collects the major changes and additions to this documentation. (If you want to know the details about why we have separate documentation for AxoSyslog, our syslog-ng distribution and how it relates to the syslog-ng documentation, read our syslog-ng documentation and similarities with AxoSyslog Core blog post.)

Version 4.7 release (2024-04-18)

For details, see the release announcement blog post.

Version 4.6 release (2024-02-01)

For details, see the release announcement blog post.

New sources

2023-10-20 to version 4.5 release (2024-01-05)

Parsers

TLS options

Manual pages

Other changes

2023-08-18 to 2023-10-20

2023-07-07 to 2023-08-18

OpenTelemetry

Other changes

New global options

2 - Introduction to AxoSyslog

This chapter introduces the AxoSyslog application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure.

What AxoSyslog is

The AxoSyslog application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging and observability solutions and telemetry pipelines, supporting on-premises, cloud-native, and hybrid environments.

Among others, AxoSyslog allows you the following.

Secure and reliable data transfer

The AxoSyslog application enables you to send log messages and other observability data from your hosts to remote servers using the latest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transfer your log messages and observability data using reliable protocols to ensure that no messages are lost.

Disk-based message buffering

To minimize the risk of losing important log messages, the AxoSyslog application can store messages on the local hard disk if the central log server or the network connection becomes unavailable. The AxoSyslog application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if AxoSyslog is restarted.

Secure logging using TLS

Log messages may contain sensitive information that should not be accessed by third parties. Therefore, AxoSyslog supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.

Flexible data extraction and processing

Most log messages are inherently unstructured, which makes them difficult to process. To overcome this problem, AxoSyslog comes with a huge set of built-in parsers, which you can combine to build very complex things.

Filter and classify

The AxoSyslog application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. You can create directories, files, and database tables dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.

Parse and rewrite

The AxoSyslog application can segment log messages to named fields or columns, and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.

To get the most information out of your log data, AxoSyslog allows you to correlate multiple log messages and aggregate the extracted information into a single message. You can also use external information to enrich your log data.

Big data clusters

The log data that your organization has to process, store, and review increases daily, so many organizations use big data solutions for their logs. To accommodate this huge amount of data, AxoSyslog natively supports storing log messages and observability data in destinations like HDFS files, Elasticsearch/OpenSearch, and cloud storage solutions like Google Pub/Sub and Amazon S3.

Message queue support

Large organizations increasingly rely on queuing infrastructure to transfer their data. For that purpose, AxoSyslog supports Apache Kafka, the Advanced Message Queuing Protocol (AMQP), and the Simple Text Oriented Messaging Protocol (STOMP).

SQL, NoSQL, and monitoring

Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The AxoSyslog application supports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, and SQLite.

AxoSyslog also allows you to extract the information you need from your log data, and directly send it to your Graphite, Redis, or Riemann monitoring system.

syslog protocol standards

AxoSyslog not only supports legacy BSD syslog (RFC3164) and the enhanced RFC5424 protocols but also JavaScript Object Notation (JSON) and journald message formats.

Heterogeneous on-premises and cloud-native environments

The AxoSyslog application is the ideal choice to collect logs and observability data in massively heterogeneous environments that use several different operating systems and hardware platforms, including Linux, Unix, BSD, Sun Solaris, HP-UX, MacOS, AIX. You can also run it in containerized environments.

IPv4 and IPv6 support

The AxoSyslog application can operate in both IPv4 and IPv6 network environments, and can receive and send messages to both types of networks.

What AxoSyslog is not

The AxoSyslog application is not log analysis software. It can filter log messages and select only the ones matching certain criteria. It can even convert the messages and restructure them to a predefined format, or parse the messages and segment them into different fields. But AxoSyslog cannot interpret and analyze the meaning behind the messages, or recognize patterns in the occurrence of different messages.

Why do you need AxoSyslog

Log messages contain information about the events happening on the hosts. Monitoring system events is essential for security and system health monitoring reasons.

The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility, and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The AxoSyslog application helps you to select only the really interesting messages, and forward them to a central server.

Company policies or other regulations often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process.

Who uses AxoSyslog

The AxoSyslog application is used worldwide by companies and institutions who collect and manage the logs of several hosts, and want to store them in a centralized, organized way. Using AxoSyslog is particularly advantageous for:

  • Internet Service Providers
  • Financial institutions and companies requiring policy compliance
  • Server, web, and application hosting companies
  • Datacenters
  • Wide area network (WAN) operators
  • Server farm administrators.

2.1 - Supported platforms

The AxoSyslog application is highly portable and is known to run on a wide range of hardware architectures (x86, x86_64, SUN Sparc, PowerPC 32 and 64, Alpha) and operating systems, including Linux, BSD, Solaris, IBM AIX, HP-UX, Mac OS X, Cygwin, and others.

3 - Quick-start guide

This chapter provides a very brief introduction into configuring the AxoSyslog application. For details on the format of the configuration file and how to configure sources, destinations, and other features, refer to the subsequent chapters.

3.1 - Configuring AxoSyslog on client hosts

Purpose:

To configure AxoSyslog on a client host, complete the following steps.

Steps:

  1. Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.

  2. Configure the local sources to collect the log messages of the host. Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.

    To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.

    • Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
    • Containers: When running AxoSyslog in a container, typically you map an external file to the /etc/syslog-ng/syslog-ng.conf file within the container. Check the mapped volumes of your container, and edit the external file.
    • Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog collector Helm chart.

    Add sources to collect the messages from your log files. File sources look like this:

        source s_myfilesource {
            file("/var/log/myapplication.log" follow-freq(1));
        };
    

    Name every source uniquely. For details on configuring file sources, see file: Collect messages from text files.

  3. Create a network destination that points directly to the AxoSyslog server, or to a local relay. The network destination greatly depends on the protocol that your log server or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:

        destination d_network { network("10.1.2.3" transport("udp")); };
    

    However, if possible, use the much more reliable IETF-syslog protocol over TCP transport:

        destination d_network {
            syslog("10.1.2.3" transport("tcp"));
        };
    
  4. Create a log statement connecting the local sources to the AxoSyslog server or relay. For example:

        log {
            source(s_local); destination(d_network);
        };
    
  5. If the logs will also be stored locally on the host, create local file destinations.

  6. Create a log statement connecting the local sources to the file destination.

  7. Set filters, macros and other features and options (for example, TLS encryption) as necessary.

Example: The default configuration file of AxoSyslog

The following is the default configuration file of AxoSyslog4.5.0. It collects local log messages and the log messages of AxoSyslog and saves them in the /var/log/messages file.

    @version: 4.5.0
    @include "scl.conf"
    source s_local {
        system(); internal();
    };
    destination d_local {
        file("/var/log/messages");
    };
    log {
        source(s_local); destination(d_local);
    };

Example: A simple configuration for clients

The following is a simple configuration file that collects local log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 4.5.0
    @include "scl.conf"
    source s_local {
        system(); internal();
    };
    destination d_syslog_tcp {
        syslog("192.168.1.1" transport("tcp") port(2010));
    };
    log {
        source(s_local);destination(d_syslog_tcp);
    };

3.2 - Configuring AxoSyslog on server hosts

Purpose:

To configure AxoSyslog on a server host, complete the following steps.

Steps:

  1. Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.

  2. Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.

  3. To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.

    • Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
    • Containers: When running AxoSyslog in a container, typically you map an external file to the /etc/syslog-ng/syslog-ng.conf file within the container. Check the mapped volumes of your container, and edit the external file.
    • Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog collector Helm chart.

    Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:

        source s_network {
            syslog(ip(10.1.2.3) transport("udp"));
        };
    

    However, if possible, use the much more reliable TCP transport:

        source s_network {
            syslog(ip(10.1.2.3) transport("tcp"));
        };
    

    For other options, see syslog: Collect messages using the IETF-syslog protocol and tcp, tcp6, udp, udp6: OBSOLETE - Collect messages from remote hosts using the BSD syslog protocol.

  4. Create local destinations that will store the log messages, for example, file- or program destinations. The default configuration of AxoSyslog places the collected messages into the /var/log/messages file:

        destination d_local {
            file("/var/log/messages");
        };
    

    If you want to create separate logfiles for every client host, use the ${HOST} macro when specifying the filename, for example:

        destination d_local {
            file("/var/log/messages_${HOST}");
        };
    

    For details on further macros and how to use them, see template and rewrite: Format, modify, and manipulate log messages.

  5. Create a log statement connecting the sources to the local destinations.

        log {
            source(s_local); source(s_network); destination(d_local);
        };
    
  6. Set filters, options (for example, TLS encryption) and other advanced features as necessary.

    Example: A simple configuration for servers

    The following is a simple configuration file for AxoSyslog that collects incoming log messages and stores them in a text file.

        @version: 4.5.0
        @include "scl.conf"
        options {
            time-reap(30);
            mark-freq(10);
            keep-hostname(yes);
        };
        source s_local {
            system(); internal();
        };
        source s_network {
            syslog(transport(tcp));
        };
        destination d_logs {
            file(
                "/var/log/syslog-ng/logs.txt"
                owner("root")
                group("root")
                perm(0777)
                );
            };
        log {
            source(s_local); source(s_network); destination(d_logs);
        };
    

3.3 - Configuring AxoSyslog relays

This section describes how to configure AxoSyslog as a relay.

3.3.1 - Configuring AxoSyslog on relay hosts

To configure AxoSyslog on a relay host, complete the following steps:

  1. Install the AxoSyslog application on the host. For details on installing AxoSyslog on specific operating systems, see Install AxoSyslog.

  2. Configure the network sources that collect the log messages sent by the clients.

  3. Create a network destination that points to the AxoSyslog server.

  4. Create a log statement connecting the network sources to the AxoSyslog server.

  5. Configure the local sources that collect the log messages of the relay host.

  6. Create a log statement connecting the local sources to the AxoSyslog server.

  7. Enable the keep-hostname() and disable the chain-hostnames() options. (For details on how these options work, see Global options.)

  8. Set filters and options (for example, TLS encryption) as necessary.

Example: A simple configuration for relays

The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 4.5.0
    @include "scl.conf"
    options {
        time-reap(30);
        mark-freq(10);
        keep-hostname(yes);
        chain-hostnames(no);
    };
    source s_local {
        system(); internal();
    };
    source s_network {
        syslog(transport(tcp));
    };
    destination d_syslog_tcp {
        syslog("192.168.1.5" transport("tcp") port(2010));
    };
    log {
        source(s_local); source(s_network);
        destination(d_syslog_tcp);
    };

3.3.2 - How relaying log messages works

Depending on your exact needs about relaying log messages, there are many scenarios and AxoSyslog options that influence how the log message will look like on the logserver. Some of the most common cases are summarized in the following example:

Consider the following example: client-host > relay > server, where the IP address of client-host is 192.168.1.2. The client-host device sends a syslog message to relay. Depending on the settings of relay, the following can happen.

  • By default, the keep-hostname() option is disabled, so relay writes the IP address of the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discarding any IP address or hostname that was originally in the message.

  • If the keep-hostname() option is enabled on relay, but name resolution is disabled (the use-dns() option is set to no), relay uses the HOST field of the message as-is, which is probably 192.168.1.2.

  • To resolve the 192.168.1.2 IP address to a hostname on relay using a DNS server, use the keep-hostname(no) and use-dns(yes) options. If the DNS server is properly configured and reverse DNS lookup is available for the 192.168.1.2 address, AxoSyslog will rewrite the HOST field of the log message to client-host.

  • The above points apply to the AxoSyslog server (server) as well, so if relay is configured properly, use the keep-hostname(yes) option on server to retain the proper HOST field. Setting keep-hostname(no) on server would result in AxoSyslog rewriting the HOST field to the address of the host that sent the message to server, which is relay in this case.

  • If you cannot or do not want to resolve the 192.168.1.2 IP address on relay, but want to store your log messages on server using the IP address of the original host (that is, client-host), you can enable the spoof-source() option on relay. However, spoof-source() works only under the following conditions:

    • The syslog-ng binary has been compiled with the --enable-spoof-source option.
    • The log messages are sent using the highly unreliable UDP transport protocol. (Extremely unrecommended.)

3.4 - Sending Kubernetes logs to OpenSearch

The following tutorial shows you how to install AxoSyslog on Kubernetes, deploy OpenSearch to the same cluster, and send Kubernetes logs to OpenSearch.

Prerequisites

You need:

  • A Kubernetes cluster. We used minikube with docker driver and Helm. We used a Ubuntu 22.04 (amd64) machine, but it should work on any system that can run minikube (2 CPUs, 2GB of free memory, 20GB of free disk space).

    The OpenSearch service needs a large mmap count setting, so set it to at least 262144, for example:

    sysctl -w vm.max_map_count=262144
    
  • Helm and kubectl installed.

Generate logs

If you don’t already have an application that generates logs deployed to the Kubernetes cluster, install kube-logging/log-generator to generate sample logs. Complete the following steps.

  1. Add the kube-logging repository to Helm.

    helm repo add kube-logging https://kube-logging.github.io/helm-charts
    

    Expected output:

    "kube-logging" has been added to your repositories
    
  2. Update your repositories.

    helm repo update
    

    The output should look like:

    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "kube-logging" chart repository
    Update Complete. ⎈Happy Helming!⎈
    
  3. Install kube-logging/log-generator.

    helm install --generate-name --wait kube-logging/log-generator
    

    The output should look like:

    NAME: log-generator-1684694629
    LAST DEPLOYED: Sun May 21 20:43:49 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    
  4. Check that the log-generator is running:

    kubectl get pods
    

    The output should look like:

    NAME                                        READY   STATUS        RESTARTS       AGE
    log-generator-1681984863-5946c559b9-ftrrn   1/1     Running       0              8s
    

Set up OpenSearch

  1. Install an OpenSearch cluster with Helm:

    helm repo add opensearch https://opensearch-project.github.io/helm-charts/
    

    Expected output:

    "opensearch" has been added to your repositories
    
  2. Update your repositories.

    helm repo update
    

    The output should look like:

    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "opensearch" chart repository
    Update Complete. ⎈Happy Helming!⎈
    
  3. Install OpenSearch. This step can take a few minutes.

    helm install --generate-name --wait opensearch/opensearch
    
  4. Install the OpenSearch dashboards.

    helm install --generate-name --wait opensearch/opensearch-dashboards
    

    The output should look like:

    NAME: opensearch-dashboards-1684695728
    LAST DEPLOYED: Sun May 21 21:02:09 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    1. Get the application URL by running these commands:
      export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch-dashboards-1684695728" -o jsonpath="{.items[0].metadata.name}")
      export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
      echo "Visit http://127.0.0.1:8080 to use your application"
      kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
    
  5. Now you should have 5 pods. Check that they exist:

    kubectl get pods
    

    The output should look like:

    NAME                                                READY   STATUS    RESTARTS   AGE
    log-generator-1681984863-5946c559b9-ftrrn           1/1     Running   0          3m39s
    opensearch-cluster-master-0                         1/1     Running   0          81s
    opensearch-cluster-master-1                         1/1     Running   0          81s
    opensearch-cluster-master-2                         1/1     Running   0          81s
    opensearch-dashboards-1681999620-59f64f98f7-bjwwh   1/1     Running   0          44s
    
  6. Forward the 5601 port of the OpenSearch Dashboards service (replace the name of the pod with your pod).

    kubectl port-forward <name-of-your-opensearch-dashboards-pod> 8080:5601
    

    The output should look like:

    Forwarding from 127.0.0.1:8080 -> 5601
    Forwarding from [::1]:8080 -> 5601
    
  7. Log in to the dashboard at http://localhost:8080 with admin/admin. You will soon create an Index Pattern here, but first you have to send some logs from syslog-ng.

Set up axosyslog-collector

  1. Add the AxoSyslog Helm repository:

    helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
    helm repo update
    
  2. Create a YAML file (called axoflow-demo.yaml in the examples) to configure the collector.

    config:
      sources:
        kubernetes:
          # Collect kubernetes logs
          enabled: true
      destinations:
        # Send logs to OpenSearch
        opensearch:
          - address: "opensearch-cluster-master"
            index: "test-axoflow-index"
            user: "admin"
            password: "admin"
            tls:
              # Do not validate the server's TLS certificate.
              peerVerify: false
            # Send the syslog fields + the metadata from .k8s.* in JSON format
            template: "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))"
    
  3. Check how the syslog-ng.conf file looks with your custom values:

    helm template -f axoflow-demo.yaml -s templates/config.yaml axosyslog/axosyslog-collector
    

    The output should look like:

    # Source: axosyslog-collector/templates/config.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      labels:
        helm.sh/chart: axosyslog-collector-0.3.0
        app.kubernetes.io/name: axosyslog-collector
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/version: "4.2.0"
        app.kubernetes.io/managed-by: Helm
      name: release-name-axosyslog-collector
    data:
      syslog-ng.conf: |
        @version: current
        @include "scl.conf"
    
        options {
          stats(
            level(1)
          );
        };
    
        log {
          source { kubernetes(); };
          destination {
            elasticsearch-http(
              url("https://opensearch-cluster-master:9200/_bulk")
              index("test-axoflow-index")
              type("")
              template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))")
              user("admin")
              password("admin")
              tls(
                peer-verify(no)
              )
            );
          };
        };    
    
  4. Install the axosyslog-collector chart:

    helm install --generate-name --wait -f axoflow-demo.yaml axosyslog/axosyslog-collector
    

    The output should look like:

    NAME: axosyslog-collector-1682002179
    LAST DEPLOYED: Thu Apr 20 16:49:39 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    1. Watch the axosyslog-collector-1682002179 container start.
    
  5. Check your pods:

    kubectl get pods --namespace=default -l app=axosyslog-collector-1682002179 -w
    kubectl get pods
    

    The output should look like:

    NAME                                                READY   STATUS    RESTARTS   AGE
    log-generator-1681984863-5946c559b9-ftrrn           1/1     Running   0          13m
    opensearch-cluster-master-0                         1/1     Running   0          11m
    opensearch-cluster-master-1                         1/1     Running   0          11m
    opensearch-cluster-master-2                         1/1     Running   0          11m
    opensearch-dashboards-1681999620-59f64f98f7-bjwwh   1/1     Running   0          10m
    axosyslog-collector-1682002179-pjlkn                1/1     Running   0          6s
    

Check the logs in OpenSearch

  1. Open OpenSearch dashboard at http://localhost:8080/app/management/opensearch-dashboards/.

  2. Create an Index Pattern called test-axoflow-index: http://localhost:8080/app/management/opensearch-dashboards/indexPatterns. At Step 2, set the Time field to @timestamp.

    OpenSearch create index pattern for syslog messages screenshot

  3. Now you can see your logs on the Discover view at http://localhost:8080/app/discover. Opening the detailed view for a log entry shows you the fields sent to OpenSearch.

    OpenSearch Kubernetes log messages collected with the AxoSyslog syslog-ng distribution Kubernetes metadata collected with the AxoSyslog syslog-ng distribution

3.5 - Managing and checking the syslog-ng service on Linux

This section describes how to start, stop and check the status of AxoSyslog service on Linux.

Starting AxoSyslog

To start AxoSyslog, execute the following command as root. For example:

systemctl start syslog-ng

If the service starts successfully, no output will be displayed.

The following message indicates that AxoSyslog can not start (see Checking AxoSyslog status):

Job for syslog-ng.service failed because the control process exited with error code. See `systemctl status syslog-ng.service` and `journalctl -xe` for details.

Stopping AxoSyslog

To stop AxoSyslog

  1. Execute the following command as root.

    systemctl stop syslog-ng

  2. Check the status of AxoSyslog service (see Checking AxoSyslog status).

Restarting AxoSyslog

To restart AxoSyslog, execute the following command as root.

systemctl restart syslog-ng

Reloading configuration file without restarting AxoSyslog

To reload the configuration file without restarting AxoSyslog, execute the following command as root.

systemctl reload syslog-ng

Checking AxoSyslog status

To check the following status-related components, observe the suggestions below.

Checking the status of AxoSyslog service

To check the status of AxoSyslog service

  1. Execute the following command as root.

    systemctl --no-pager status syslog-ng

  2. Check the Active: field, which shows the status of AxoSyslog service. The following statuses are possible:

    • active (running) - syslog-ng service is up and running

          syslog-ng.service - System Logger Daemon
          Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
          Active: active (running) since Tue 2019-06-25 08:58:09 CEST; 5s ago
          Main PID: 6575 (syslog-ng)
          Tasks: 3
          Memory: 13.3M
          CPU: 268ms
          CGroup: /system.slice/syslog-ng.service
          6575 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
      
    • inactive (dead) - syslog-ng service is stopped

          syslog-ng.service - System Logger Daemon
          Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
          Active: inactive (dead) since Tue 2019-06-25 09:14:16 CEST; 2min 18s ago
          Process: 6575 ExecStart=/opt/syslog-ng/sbin/syslog-ng -F --no-caps --enable-core $SYSLOGNG_OPTIONS (code=exited, status=0/SUCCESS)
          Main PID: 6575 (code=exited, status=0/SUCCESS)
          Status: "Shutting down... Tue Jun 25 09:14:16 2019"
          Jun 25 09:14:31 as-syslog-srv systemd: Stopped System Logger Daemon.
      

Checking the process of AxoSyslog

To check the process of AxoSyslog, execute one of the following commands.

  • ps u <pid of syslog-ng>

    Expected output example:

    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
    
  • ps axu | grep syslog-ng | grep -v grep

    Expected output example:

    syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
    

Checking the internal logs of AxoSyslog

The internal logs of AxoSyslog contains informal, warning and error messages.

By default, AxoSyslog log messages (generated on the internal() source) are written to /var/log/messages.

Check the internal logs of AxoSyslog for any issue.

Message processing

The AxoSyslog application collects statistics about the number of processed messages on the different sources and destinations.

Central statistics

To check the central statistics, execute the following command to see the number of received and queued (sent) messages by AxoSyslog.

watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center"

The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is processing the messages. Output example:

    Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center       Tue Jun 25 10:33:25 2019
    center;;queued;a;processed;112
    center;;received;a;processed;28

Source statistics

To check the source statistics, execute the following command to see the number of received messages on the configured sources.

watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source"

The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is receiving messages on the sources. Output example:

    Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source      Tue Jun 25 10:40:50 2019
    source;s_null;;a;processed;0
    source;s_net;;a;processed;0
    source;s_local;;a;processed;90

Destination statistics

To check the source statistics, execute the following command to see the number of received messages on the configured sources.

watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source"

The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is receiving messages on the sources. Output example:

    Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^destination      Tue Jun 25 10:41:02 2019
    destination;d_logserver2;;a;processed;90
    destination;d_messages;;a;processed;180
    destination;d_logserver;;a;processed;90
    destination;d_null;;a;processed;0

4 - Install AxoSyslog

This chapter explains how to install AxoSyslog on various platforms.

Cloud-ready syslog-ng images

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

  • They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
  • They incorporate cloud-native features and settings, such as the Kubernetes source.
  • They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
  • They support the ARM architecture.

The AxoSyslog images support the following architectures:

  • amd64
  • arm/v7
  • arm64

Install AxoSyslog

Other installation methods

  • You can install AxoSyslog on many platforms using the package manager and official repositories of the platform. For a list of third-party packages available for various Linux, UNIX, and other platforms, see syslog-ng Open Source Edition installation packages.
  • For instructions on compiling syslog-ng Open Source Edition from the source code, see the GitHub project page.

4.1 - Install AxoSyslog with Podman and systemd

This page shows you how to run AxoSyslog as a systemd service using podman.

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

  • They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
  • They incorporate cloud-native features and settings, such as the Kubernetes source.
  • They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
  • They support the ARM architecture.

The AxoSyslog images support the following architectures:

  • amd64
  • arm/v7
  • arm64

Prerequisites

Podman version 4.6.1.

The steps in this procedure were tested on CentOS 9, but should work on other similar distributions as well.

Install AxoSyslog as a systemd service

  1. Make sure that there is no axosyslog.service unit file on the system. Run the following commands:

    sudo rm /etc/systemd/system/axosyslog.service
    

    Expected output:

    rm: cannot remove '/etc/systemd/system/axosyslog.service': No such file or directory
    
    sudo systemctl cat axosyslog.service
    

    Expected output:

    No files found for axosyslog.service.
    
  2. Create a systemd unit file called /etc/containers/systemd/axosyslog.container based on the following template:

    sudo curl -o /etc/containers/systemd/axosyslog.container https://axoflow.com/docs/axosyslog-core/install/podman-systemd/axosyslog.container
    
        
  3. Edit the unit file as needed for your environment.

    We recommend using the default mount points:

    PurposeOn the hostIn the container
    Disk-buffer and persist files/var/lib/syslog-ng/var/lib/syslog-ng
    syslog-ng configuration file/opt/axosyslog/etc/etc/syslog-ng
    Output log files/opt/axosyslog/var/log/var/log
  4. (Optional) Create an override.conf file to set custom environment values. This can be useful if you don’t want to modify /etc/containers/systemd/axosyslog.container. Run:

    systemctl edit axosyslog
    

    Later you can edit this file by running the previous command again.

  5. Create the /opt/axosyslog/etc/syslog-ng.conf configuration file based on the following template.

    sudo mkdir -p /opt/axosyslog/etc/ ; sudo curl -o /opt/axosyslog/etc/syslog-ng.conf https://axoflow.com/docs/axosyslog-core/install/podman-systemd/syslog-ng.conf
    

    With the following sample configuration file AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages file.

        

    You can customize the configuration file according to your needs. For a few pointers, see Configuring AxoSyslog on server hosts and the rest of this guide.

  6. Run the following commands to reload the systemd configuration and launch the axosyslog service. Though the systemctl commands are run as root, the container will run as the specified user if set appropriately in the unit file.

    sudo systemctl daemon-reload
    sudo systemctl stop axosyslog
    sudo systemctl start axosyslog
    

    If there aren’t any errors, these commands don’t have any output.

  7. Run the following command to verify that the service was properly started:

    journalctl -b -u axosyslog | tail -100
    

    The output should be similar to:

    Feb 12 09:04:40 <your-hostname> systemd[1]: Starting AxoSyslog Container...
    Feb 12 09:04:40 <your-hostname> podman[2783]: 2024-02-12 09:04:40.454665314 -0500 EST m=+0.167732500 system refresh
    Feb 12 09:04:40 <your-hostname> axosyslog[2783]: Trying to pull ghcr.io/axoflow/axosyslog:latest...
    Feb 12 09:04:40 <your-hostname> axosyslog[2783]: Pulling image //ghcr.io/axoflow/axosyslog:latest inside systemd: setting pull timeout to 5m0s
    Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Getting image source signatures
    Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
    Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:619be1103602d98e1963557998c954c892b3872986c27365e9f651f5bc27cab8
    Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:b061f41886afb563aff2a5f731f3286ba54ea6f657ed3e282f5339a12a64c5ef
    Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:1b8d965a650c6a05227bd5c549930c9898071e8e7abb26886d4169a99762de0a
    Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:b5b0ce6ebef193c4f909379188cfb59443e8a1809816fbb476074908b170b4d1
    Feb 12 09:04:50 <your-hostname> axosyslog[2783]: Copying config sha256:c379d94ef2c5ec348dfb3a93eed9a19aed667c396008db85edc354c8f4f8cb6a
    Feb 12 09:04:50 <your-hostname> axosyslog[2783]: Writing manifest to image destination
    Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.422390687 -0500 EST m=+10.135457863 container create 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.url=https://axoflow.io/, org.opencontainers.image.source=https://github.com/axoflow/axosyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.title=AxoSyslog, org.opencontainers.image.vendor=Axoflow, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, maintainer=axoflow.io, org.opencontainers.image.licenses=GPL-3.0-only)
    Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.402626446 -0500 EST m=+10.115693622 image pull c379d94ef2c5ec348dfb3a93eed9a19aed667c396008db85edc354c8f4f8cb6a ghcr.io/axoflow/axosyslog:latest
    Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.489925509 -0500 EST m=+10.202992695 container init 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.licenses=GPL-3.0-only, org.opencontainers.image.vendor=Axoflow, maintainer=axoflow.io, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.url=https://axoflow.io/, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.title=AxoSyslog, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, org.opencontainers.image.source=https://github.com/axoflow/axosyslog)
    Feb 12 09:04:50 <your-hostname> systemd[1]: Started AxoSyslog Container.
    Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.500050669 -0500 EST m=+10.213117845 container start 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.source=https://github.com/axoflow/axosyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.licenses=GPL-3.0-only, org.opencontainers.image.vendor=Axoflow, org.opencontainers.image.title=AxoSyslog, maintainer=axoflow.io, org.opencontainers.image.url=https://axoflow.io/)
    Feb 12 09:04:50 <your-hostname> axosyslog[2783]: 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f
    Feb 12 09:04:50 <your-hostname> AxoSyslog[2821]: [2024-02-12T14:04:50.806054] syslog-ng starting up; version='4.6.0'
    
  8. Send a test message to the service:

    echo '<5> localhost test: this is a test message' | nc localhost 514
    

    Check that the test message has arrived into the log file:

    less /opt/axosyslog/var/log/messages
    

    The output should be similar to:

    Feb 19 15:49:12 localhost test: this is a test message
    

Customize the configuration

To customize the configuration, edit the /opt/axosyslog/etc/syslog-ng.conf file on the host, then reload the service.

Managing the AxoSyslog systemd service

  • You can reload syslog-ng running in the container via systemctl. The following command reloads the syslog-ng.conf file, without stopping/starting syslog-ng itself.

    sudo systemctl reload axosyslog
    
  • You can access syslog-ng-ctl from the host, for example by running:

    podman exec -ti AxoSyslog syslog-ng-ctl show-license-info
    

    If you use syslog-ng-ctl regularly, you can create the /opt/axosyslog/bin/syslog-ng-ctl file with the following content, make it executable, and add it to your path. That way running syslog-ng-ctl <command> will execute the command in the AxoSyslog container.

    #!/bin/bash  
    
    podman exec -ti AxoSyslog syslog-ng-ctl "$@"  
    
  • The traditional method of starting a service at boot (systemctl enable) is not supported for container services. To automatically start the AxoSyslog service, make sure that the following line is included in the unit file. (It is included in the sample template.)

    [Install]
    WantedBy=default.target
    

4.2 - Install AxoSyslog with Helm

AxoSyslog provides Helm charts for syslog-ng. You can use these charts to install cloud-ready syslog-ng images created and maintained by Axoflow.

Prerequisites

You must have Helm 3.0 or newer installed to use these charts. Refer to the official Helm documentation for details.

Limitations

The chart provides parameters that make it easy to:

  • collect logs using the kubernetes() source, and
  • forward the logs using the network() and opensearch() destinations.

To use other sources and destinations, use the config.raw parameter. For the list of configurable parameters and their default values, see Parameters of the AxoSyslog collector Helm chart.

Install

To install the axosyslog-collector charts, complete the following steps.

  1. Clone the chart repository.

    helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
    helm repo update
    
  2. Install the chart. The following command installs axosyslog-collector into the default namespace. For the list of configurable parameters and their default values, see Parameters of the AxoSyslog collector Helm chart. If you want to use disk-buffers, see also How to use disk-buffers in containers and Kubernetes.

    helm install --generate-name axosyslog/axosyslog-collector
    
    NAME: axosyslog-collector-1683469360
    LAST DEPLOYED: Sun May  7 16:22:40 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    1. Watch the axosyslog-collector-1683469360 container start.
      $ kubectl get pods --namespace=default -l app=axosyslog-collector-1683469360 -w
    
  3. Check that the pod is running.

    kubectl get pods
    

    The output should look like:

    NAME                                   READY   STATUS    RESTARTS   AGE
    axosyslog-collector-1683469360-tptfb   1/1     Running   0          28s
    

How to use disk-buffers in containers and Kubernetes

When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.

  • Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
    • In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
    • In a container, mount the disk-buffer directory from the host, or store it on a local volume.
  • Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
  • Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.

If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.

Uninstall

Tip: List all installed releases using helm list.

To uninstall a chart release, run:

helm delete <name-of-the-release-to-delete>

4.2.1 - Parameters of the AxoSyslog collector Helm chart

The following table lists the configurable parameters of the AxoSyslog collector chart and their default values. For details on installing the chart, see Install AxoSyslog with Helm.

Parameters for syslog-ng configuration

ParameterDescriptionDefault
config.rawA complete syslog-ng configuration. If this parameter is set, all other parameters in the config section are ignored. For details on how to create a configuration for syslog-ng, see the AxoSyslog Core documentation.""
config.versionThe version string specifies the syslog-ng version the configuration corresponds to.""
config.sources.kubernetes.enabledCollect pod logs using the kubernetes() source. If disabled, the chart doesn’t configure any source. For the list of available sources, see the (/docs/axosyslog-core/chapter-sources/)true

The following example uses the config.raw parameter to configure a custom destination:

config:
  raw: |
    @version: 4.5.0
    @include "scl.conf"

    log {
      source {
        syslog(port(12345));
      };

      destination {
        logscale(
          token("your-secret-humio-ingest-token")
        );
      };

      flags(flow-control);
    };

daemonset:
  hostNetworking: true

Network destination

Send logs over the network, conforming to RFC3164 using the network() destination.

ParameterDescriptionDefault
config.destination.network.addressThe IP address of the destination host.""
config.destination.network.transportThe transport protocol to use. Possible values: tcp, udp""
config.destination.network.portThe port number to send the messages to.""
config.destination.network.templateA template to format the messages.""

For example:

config:
  destinations:
    network:
      - transport: tcp
        address: localhost
        port: 12345
        template: "$(format-json .*)"

OpenSearch destination

Send logs to OpenSearch over HTTP or HTTPS.

ParameterDescriptionDefault
config.destination.opensearch.addressThe IP address of the OpenSearch server.""
config.destination.opensearch.indexName of the OpenSearch index that stores the messages.""
config.destination.opensearch.userThe username to use for authentication on the OpenSearch server, if not authenticating with a certificate.""
config.destination.opensearch.passwordThe password to use for authentication on the OpenSearch server.""
config.destination.opensearch.tls.CAFileThe CA certificate in PEM format to use when verifying the certificate of the server.""
config.destination.opensearch.tls.CADirA directory containing a set of trusted CA certificates in PEM format. The name of the files must be the 32-bit hash of the subject’s name. AxoSyslog collector verifies the certificate of the server using these CA certificates.""
config.destination.opensearch.tls.CertName of a file containing an X.509 certificate or a certificate chain in PEM format. AxoSyslog collector authenticates with this certificate on the server, with the private key set in the config.destination.opensearch.tls.Key field. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.""
config.destination.opensearch.tls.KeyName of a file containing an unencrypted private key in PEM format. AxoSyslog collector authenticates with this key and the certificate set in the config.destination.opensearch.tls.Cert field.""
config.destination.opensearch.tls.peerVerifyIf true, AxoSyslog collector verifies the certificate of the server with the CA certificates set in config.destination.opensearch.tls.CAFile and config.destination.opensearch.tls.CADir.""
config.destination.opensearch.templateA template to format the messages.""

For example:

config:
  destinations:
    opensearch:
      - address: 10.104.232.94
        index: "test-axoflow-index"
        tls:
          CAFile: "/path/to/CAFile.pem"
          CADir: "/path/to/CADir/"
          Cert: "/path/to/Cert.pem"
          Key: "/path/to/Key.pem"
          peerVerify: true
          template: "$(format-json .*)"

Generic parameters

ParameterDescriptionDefault
image.repositoryThe container image repositoryghcr.io/axoflow/axosyslog
image.pullPolicyThe container image pull policyIfNotPresent
image.tagThe container image tag4.5.0
image.extraArgsCustom arguments applied as the value of spec.container.args[]
imagePullSecretsThe names of secrets containing private registry credentials[]
nameOverrideOverride the chart name""
fullnameOverrideOverride the full chart name""
daemonset.enabledDeploy AxoSyslog as a DaemonSettrue
daemonset.labelsAdditional labels to apply to the DaemonSet{}
daemonset.annotationsAdditional annotations to apply to the DaemonSet{}
daemonset.affinityPod affinity{}
daemonset.nodeSelectorNode labels for pod assignment{}
daemonset.resourcesResource requests and limits{}
daemonset.tolerationsTolerations for pod assignment[]
daemonset.hostAliasesAdd host aliases[]
daemonset.secretMountsMount additional secrets as volumes[]
daemonset.extraVolumesAdditional volumes to mount[]
daemonset.securityContextSecurity context for the pod{}
daemonset.maxUnavailableThe maximum number of unavailable pods during a rolling update1
daemonset.hostNetworkingWhether to enable host networkingfalse
rbac.createWhether to create RBAC resourcesfalse
rbac.extraRulesAdditional RBAC rules[]
openShift.enabledWhether to deploy on OpenShiftfalse
openShift.securityContextConstraints.createWhether to create SecurityContextConstraints on OpenShifttrue
openShift.securityContextConstraints.annotationsAnnotations to apply to SecurityContextConstraints{}
serviceAccount.createWhether to create a service accountfalse
serviceAccount.annotationsAnnotations to apply to the service account{}
namespaceThe Kubernetes namespace to deploy to""
podAnnotationsAdditional annotations to apply to the pod{}
podSecurityContextSecurity context for the pod{}
securityContextSecurity context for the container{}
resourcesResource requests and limits for the container. If not set, the values of daemonset.resources are used.{}
nodeSelectorNode labels for pod assignment{}
tolerationsTolerations for pod assignment[]
affinityPod affinity{}
updateStrategyUpdate strategy for the DaemonSetRollingUpdate
kubernetes.enabledEnable kubernetes log collectiontrue
kubernetes.prefixSet JSON prefix for logs collected from the k8s cluster""
kubernetes.keyDelimiterSet JSON key delimiter for logs collected from the k8s cluster""
priorityClassNameThe name of the PriorityClass the pod belongs to""
dnsConfigThe DNS configuration of the pod{}
hostAliasesAdditional entries to the pod’s hosts file[]
secretMountsAdditional secrets to mount for the pod. If not set, the values of daemonset.secretMounts are used.[]
extraVolumesAdditional volumes to mount for the pod. If not set, the values of daemonset.extraVolumes are used.[]
terminationGracePeriodSecondsHow many seconds a pod with a failing probe has before shut down30

4.3 - Install AxoSyslog with Podman

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

  • They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
  • They incorporate cloud-native features and settings, such as the Kubernetes source.
  • They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
  • They support the ARM architecture.

The AxoSyslog images support the following architectures:

  • amd64
  • arm/v7
  • arm64

Install the AxoSyslog images

You can find the list of tagged versions at https://github.com/axoflow/axosyslog-docker/pkgs/container/axosyslog.

To install the latest stable version, run:

podman pull ghcr.io/axoflow/axosyslog:latest

You can also use it as a base image in your Dockerfile:

FROM ghcr.io/axoflow/axosyslog:latest

If you want to test a development version, you can use the nightly builds:

podman pull ghcr.io/axoflow/axosyslog:nightly

Note: These named packages are automatically updated when a new package is released. To install a specific version, run podman pull ghcr.io/axoflow/axosyslog:<version-number>, for example:

podman pull ghcr.io/axoflow/axosyslog:4.5.0

Customize the configuration

The AxoSyslog container image stores the configuration file at /etc/syslog-ng/syslog-ng.conf. By default, AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages and /var/log/messages-kv.log files using this configuration file from the syslog-ng repository.

To customize the configuration, create your own configuration file and override the file in the container image with it, for example:

podman run --rm --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

How to use disk-buffers in containers and Kubernetes

When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.

  • Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
    • In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
    • In a container, mount the disk-buffer directory from the host, or store it on a local volume.
  • Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
  • Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.

If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.

Expose port to receive incoming traffic

To receive incoming network in a container, you must expose the port from the container where you want to receive the traffic to the host that’s running the container. Typically, this is only needed if you are running AxoSyslog as a relay or a server/aggregator.

By default, the AxoSyslog container images expose the ports commonly used to receive syslog traffic:

  • 514/udp, typically used for RFC3164 (BSD-syslog) formatted traffic.
  • 601/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic.
  • 6514/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic over TLS.

To expose a specific port, use the --expose option when starting the container. Make sure to include the IP address of the host to make the port externally accessible.

For example, if you are receiving OpenTelemetry messages using the opentelemetry() source, expose the 4317 port:

podman run --rm --expose 127.0.0.1:4317:4317/tcp --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

4.4 - Install AxoSyslog with Docker

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

  • They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
  • They incorporate cloud-native features and settings, such as the Kubernetes source.
  • They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
  • They support the ARM architecture.

The AxoSyslog images support the following architectures:

  • amd64
  • arm/v7
  • arm64

Install the AxoSyslog images

You can find the list of tagged versions at https://github.com/axoflow/axosyslog-docker/pkgs/container/axosyslog.

To install the latest stable version, run:

docker pull ghcr.io/axoflow/axosyslog:latest

You can also use it as a base image in your Dockerfile:

FROM ghcr.io/axoflow/axosyslog:latest

If you want to test a development version, you can use the nightly builds:

docker pull ghcr.io/axoflow/axosyslog:nightly

Note: These named packages are automatically updated when a new package is released. To install a specific version, run docker pull ghcr.io/axoflow/axosyslog:<version-number>, for example:

docker pull ghcr.io/axoflow/axosyslog:4.5.0

Customize the configuration

The AxoSyslog container image stores the configuration file at /etc/syslog-ng/syslog-ng.conf. By default, AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages and /var/log/messages-kv.log files using this configuration file from the syslog-ng repository.

To customize the configuration, create your own configuration file and override the file in the container image with it, for example:

docker run --rm --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

How to use disk-buffers in containers and Kubernetes

When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.

  • Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
    • In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
    • In a container, mount the disk-buffer directory from the host, or store it on a local volume.
  • Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
  • Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.

If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.

Expose port to receive incoming traffic

To receive incoming network in a container, you must expose the port from the container where you want to receive the traffic to the host that’s running the container. Typically, this is only needed if you are running AxoSyslog as a relay or a server/aggregator.

By default, the AxoSyslog container images expose the ports commonly used to receive syslog traffic:

  • 514/udp, typically used for RFC3164 (BSD-syslog) formatted traffic.
  • 601/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic.
  • 6514/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic over TLS.

To expose a specific port, use the --expose option when starting the container. Make sure to include the IP address of the host to make the port externally accessible.

For example, if you are receiving OpenTelemetry messages using the opentelemetry() source, expose the 4317 port:

docker run --rm --expose 127.0.0.1:4317:4317/tcp --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

5 - The concepts of AxoSyslog

This chapter discusses the technical concepts of AxoSyslog.

5.1 - The philosophy of AxoSyslog

Typically, AxoSyslog is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices — called AxoSyslog clients — all run AxoSyslog, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote AxoSyslog server, which sorts and stores them.

5.2 - Logging with AxoSyslog

The AxoSyslog application reads incoming messages and forwards them to the selected destinations. The AxoSyslog application can receive messages from files, remote hosts, and other sources.

Log messages enter AxoSyslog in one of the defined sources, and are sent to one or more destinations.

Sources and destinations are independent objects, log paths define what AxoSyslog does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations: messages arriving from a source are sent to every destination listed in the log path. A log path defined in AxoSyslog is called a log statement.

Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, AxoSyslog sends only the messages satisfying the filter rules to the destinations set in the log path.

Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.

5.2.1 - The route of a log message in AxoSyslog

Purpose:

The following procedure illustrates the route of a log message from its source on the AxoSyslog client to its final destination on the central AxoSyslog server.

The route of a log message

Steps:

  1. A device or application sends a log message to a source on the AxoSyslog client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file.

  2. The AxoSyslog client running on the web server reads the message from its /var/log/apache source.

  3. The AxoSyslog client processes the first log statement that includes the /var/log/apache source.

  4. The AxoSyslog client performs optional operations (message filtering, parsing, and rewriting) on the message, for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, AxoSyslog sends the message to the destinations set in the log statement, for example, to the remote AxoSyslog server.

  5. The AxoSyslog client processes the next log statement that includes the /var/log/apache source, repeating Steps 3-4.

  6. The message sent by the AxoSyslog client arrives from a source set in the AxoSyslog server.

  7. The AxoSyslog server reads the message from its source and processes the first log statement that includes that source.

  8. The AxoSyslog server performs optional operations (message filtering, parsing, and rewriting) on the message, for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, AxoSyslog sends the message to the destinations set in the log statement.

  9. The AxoSyslog server processes the next log statement, repeating Steps 7-9.

5.3 - Modes of operation

The AxoSyslog application has three typical operation scenarios: Client, Server, and Relay.

5.3.1 - Client mode

Processing logs in client mode

In client mode, AxoSyslog collects the local logs generated by the host and forwards them through a network connection to the central AxoSyslog server or to a relay. Clients often also log the messages locally into files.

5.3.2 - Relay mode

Processing logs in relay mode

In relay mode, AxoSyslog receives logs through the network from AxoSyslog clients and forwards them to the central AxoSyslog server using a network connection. Relays also log the messages from the relay host into a local file, or forward these messages to the central AxoSyslog server.

Example relay use cases

The relay collects log messages through the network and after processing, but without writing them on the disk for storage, forwards them to one or more remote destinations.

You can use a relay for many different use cases as described in the examples below.

UDP-only source devices

Most network devices send log messages over UDP. However, UDP does not guarantee that all packets are delivered, which makes UDP unreliable.

To ensure at least a best effort level of reliability, Axoflow recommends that you deploy a relay on the network, close to the source devices. With the most reliable hops between the source and the relay, you can minimize the risk of losing UDP packets. Once the packet arrives at the relay, AxoSyslog ensures that the messages are delivered to the central server in a reliable manner.

Too many source devices

Depending on the hardware and configuration, an average instance can usually handle the following number of concurrent connections:

If the maximum message rate is lower than 200,000 messages per second:

  • maximum ca. 5,000 TCP connections

  • maximum ca. 1,000 TLS connections

If the message rate is higher than 200,000 messages per second, contact Axoflow.

If you have more source devices, you must deploy a relay machine at least per 5,000 sources and batch up all the logs into a single TCP connection that connects the relay to the server. If TLS is used, deploy relays per 1,000 source devices.

Collecting logs from remote sites (especially over public WAN)

If you need to collect log messages from geographically remote sites or over public WAN, Axoflow recommends that you install at least a relay node per each remote site. The relay can be the last outgoing hop for all the messages of the remote site, which has several benefits:

  • Maintenance: You only need to change the configuration of the relay if you want to re-route the logs of some or all sources of the remote site. Also you do not need to change each source’s configuration one by one.

  • Security: If you trust your internal network, it is not necessary to hold encrypted connections within the LAN of the remote site as the messages can get to the relay without encryption. Messages must be sent in an encrypted way over the public WAN, and it is enough to hold only a single TCP/TLS connection between the sites, that is, between the remote relay and the central server. This eliminates the wasting of resources as holding several TLS connections directly from the clients is more costly than holding a single connection from the relay.

  • Reliability: You can set up a main disk-buffer on the relay. The main disk-buffer is only responsible for buffering all the logs of the remote site if the central AxoSyslog server is temporarily unavailable. It is easier to maintain this single main disk-buffer instead of setting disk-buffers on individual client machines.

Separation, distribution, and balancing of message processing tasks

Most Linux applications have their own human readable, but difficult to handle, log messages. Without parsing and normalization it is difficult to alert and report on these log messages. Many AxoSyslog users use the message parsing tools of AxoSyslog to normalize their different log messages. Just like normalization, filtering can also be resource-heavy, depending on what the filtering is based on. In this case, it might be inefficient to perform all the message processing tasks on the server as it can result in decreased overall performance.

It is a typical setup to deploy relays in front of the central server operating as a receiver front-end. Most resource-heavy tasks, for example, parsing, filtering, and so on, are performed on this receiver layer. As all resource-heavy tasks are performed on the relay, the central server behind it only needs to get the messages from the relay and write them into the final text-based or tamper-proof (logstore) format. Since you can run several relays, you can balance the resource-heavy tasks between more relays, and a single server behind the relays can still be fast enough to write all the messages on the disk.

Acting as a relay also depends on the functionality. A relay does not have to be a dedicated relay machine at all. For log collection, it can be one of the clients with a relay configuration. Note that in a robust log collection infrastructure, the relays have their own purpose, and Axoflow recommends running dedicated relay machines.

You can run several parallel relays to ensure horizontal redundancy. For example, if each of the relays has the same configuration, when one relay goes down another relay can take over the processing. Distribution of the logs can be done by the built-in client-side failover functionality and also by a general load balancer. The load balancer is also used to serve N+1 redundant relay deployments. In this case, switching from one relay to another relay is done when there is an outage but also for real load balancing purposes.

What AxoSyslog relays are not good for

The purpose of the relay is to buffer the logs for short term, for example, a few minutes or a few hours long outages (depending on the log volume). It is not designed to buffer logs generated by the sources during a very long server or connection outage, for example, up to a few days long.

If you expect extended outages, Axoflow recommends that you deploy servers instead of relays. There are many deployments where long term storage and archiving are performed on the central AxoSyslog server, but relays also do short-term log storage.

5.3.3 - Server mode

Processing logs in server mode

In server mode, AxoSyslog acts as a central log-collecting server. It receives messages from AxoSyslog clients and relays over the network, and stores them locally in files, or passes them to other applications, for example, log analyzers.

5.4 - Global objects

The AxoSyslog application uses the following objects:

  • Source driver: A communication method used to receive log messages. For example, AxoSyslog can receive messages from a remote host via TCP/IP, or read the messages of a local application from a file. For details on source drivers, see source: Read, receive, and collect log messages.

  • Source: A named collection of configured source drivers.

  • Destination driver: A communication method used to send log messages. For example, AxoSyslog can send messages to a remote host via TCP/IP, or write the messages into a file or database. For details on destination drivers, see destination: Forward, send, and store log messages.

  • Destination: A named collection of configured destination drivers.

  • Filter: An expression to select messages. For example, a simple filter can select the messages received from a specific host. For details, see Customize message format using macros and templates.

  • Macro: An identifier that refers to a part of the log message. For example, the ${HOST} macro returns the name of the host that sent the message. Macros are often used in templates and filenames. For details, see Customize message format using macros and templates.

  • Parser: Parsers are objects that parse the incoming messages, or parts of a message. For example, the csv-parser() can segment messages into separate columns at a predefined separator character (for example, a comma). Every column has a unique name that can be used as a macro. For details, see parser: Parse and segment structured messages and db-parser: Process message content with a pattern database (patterndb).

  • Rewrite rule: A rule modifies a part of the message, for example, replaces a string, or sets a field to a specified value. For details, see Modifying messages using rewrite rules.

  • Log paths: A combination of sources, destinations, and other objects like filters, parsers, and rewrite rules. The AxoSyslog application sends messages arriving from the sources of the log paths to the defined destinations, and performs filtering, parsing, and rewriting of the messages. Log paths are also called log statements. Log statements can include other (embedded) log statements and junctions to create complex log paths. For details, see log: Filter and route log messages using log paths, flags, and filters.

  • Template: A template is a set of macros that can be used to restructure log messages or automatically generate file names. For example, a template can add the hostname and the date to the beginning of every log message. For details, see Customize message format using macros and templates.

  • Option: Options set global parameters of syslog-ng, like the parameters of name resolution and timezone handling. For details, see Global options.

For details on the above objects, see The configuration syntax in detail.

5.5 - Timezones and daylight saving

The AxoSyslog application receives the timezone and daylight saving information from the operating system it is installed on. If the operating system handles daylight saving correctly, so does syslog-ng.

The AxoSyslog application supports messages originating from different timezones. The original syslog protocol (RFC3164) does not include timezone information, but syslog-ng provides a solution by extending the syslog protocol to include the timezone in the log messages. The AxoSyslog application also enables administrators to supply timezone information for legacy devices which do not support the protocol extension.

5.5.1 - Assigning timezone to the message

When AxoSyslog receives a message, it assigns timezone information to the message using the following algorithm.

  1. The sender application or host specifies the timezone of the messages. If the incoming message includes a timezone it is associated with the message. Otherwise, the local timezone is assumed.

  2. Specify the time-zone() parameter for the source driver that reads the message. This timezone will be associated with the messages only if no timezone is specified within the message itself. Each source defaults to the value of the recv-time-zone() global option. It is not possible to override only the timezone information of the incoming message, but setting the keep-timestamp() option to no allows AxoSyslog to replace the full timestamp (timezone included) with the time the message was received.

  3. Specify the timezone in the destination driver using the time-zone() parameter. Each destination driver might have an associated timezone value: syslog-ng converts message timestamps to this timezone before sending the message to its destination (file or network socket). Each destination defaults to the value of the send-time-zone() global option.

  4. If the timezone is not specified, local timezone is used.

  5. When macro expansions are used in the destination filenames, the local timezone is used. (Also, if the timestamp of the received message does not contain the year of the message, AxoSyslog uses the local year.)

5.5.2 - A note on timezones and timestamps

If the clients run syslog-ng, then use the ISO timestamp, because it includes timezone information. That way you do not need to adjust the recv-time-zone() parameter of syslog-ng.

If you want to output timestamps in Unix (POSIX) time format, use the S_UNIXTIME and R_UNIXTIME macros. You do not need to change any of the timezone related parameters, because the timestamp information of incoming messages is converted to Unix time internally, and Unix time is a timezone-independent time representation. (Actually, Unix time measures the number of seconds elapsed since midnight of Coordinated Universal Time (UTC) January 1, 1970, but does not count leap seconds.)

5.6 - Product licensing

Starting with version 3.2, the AxoSyslog application is licensed under a combined LGPL+GPL license. The core of AxoSyslog is licensed under the GNU Lesser General Public License Version 2.1 license, while the rest of the codebase is licensed under the GNU General Public License Version 2 license.

For details about the LGPL and GPL licenses, see GNU Lesser General Public License and GNU General Public License, respectively.

The Documentation is licensed separately. For details, see The syslog-ng OSE Documentation License.

5.7 - High availability support

Multiple AxoSyslog servers can be run in fail-over mode. The AxoSyslog application does not include any internal support for this, as clustering support must be implemented on the operating system level. A tool that can be used to create UNIX clusters is Heartbeat (for details, see this page).

Starting with AxoSyslog version 3.2, AxoSyslog clients can be configured to send the log messages to failover servers in case the primary syslog server becomes unaccessible. For details on configuring failover servers, see the description of the failover-servers() destination option in destination: Forward, send, and store log messages.

5.8 - The structure of a log message

The following sections describe the structure of log messages. Currently there are two standard syslog message formats:

5.8.1 - BSD-syslog or legacy-syslog messages

This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol. A syslog message consists of the following parts:

  • [PRI](/docs/axosyslog-core/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-pri/)

  • [HEADER](/docs/axosyslog-core/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-header/)

  • [MSG](/docs/axosyslog-core/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-msg/)

The total message cannot be longer than 1024 bytes.

The following is a sample syslog message:

   <133>Feb 25 14:09:07 webserver syslogd: restart

The message corresponds to the following format:

   <priority>timestamp hostname application: message

The different parts of the message are explained in the following sections.

5.8.1.1 - The PRI message part

This section describes the PRI message part of a syslog message, according to the legacy-syslog or BSD-syslog protocol.

For further details about the HEADER and MSG parts of a syslog message, see the following sections:

  • [HEADER](/docs/axosyslog-core/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-header/)

  • [MSG](/docs/axosyslog-core/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-msg/)

The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while Severity marks its importance.

PRI formula

The Priority value is calculated using the following formula:

   <PRI> = ( <facility> * 8) + <severity> 

That is, you first multiply the Facility number by 8, and then add the numerical value of the Severity to the multiplied sum.

Example: the correlation between facility value, severity value, and the Priority value in the PRI message part

The following example illustrates a sample syslog message with a sample PRI field (that is, Priority value):

   <133> Feb 25 14:09:07 webserver syslogd: restart

In this example, <133> represents the PRI field (Priority value). The syslog message’s Facility value is 16, and the Severity value is 5.

Substituting the numerical values into the <PRI> = ( <facility> * 8) + <severity> formula, the results match the Priority value in our example:

<133> = ( <16> * 8) + <5>.

Facility and Severity values

The possible Facility values (between 0 and 23) and Severity values (between 0 and 7) each correspond to a message type (see Table 1: syslog Message Facilities), or a message importance level (see Table 2: syslog Message Severities).

syslog Message Facilities

The following table lists possible Facility values.

Numerical CodeFacility
0kernel messages
1user-level messages
2mail system
3system daemons
4security/authorization messages
5messages generated internally by syslogd
6line printer subsystem
7network news subsystem
8UUCP subsystem
9clock daemon
10security/authorization messages
11FTP daemon
12NTP subsystem
13log audit
14log alert
15clock daemon
16-23locally used facilities (local0-local7)

syslog Message Severities

The following table lists possible Severity values.

Numerical CodeSeverity
0Emergency: system is unusable
1Alert: action must be taken immediately
2Critical: critical conditions
3Error: error conditions
4Warning: warning conditions
5Notice: normal but significant condition
6Informational: informational messages
7Debug: debug-level messages

5.8.1.2 - The HEADER message part

This section describes the HEADER message part of a syslog message, according to the legacy-syslog or BSD-syslog protocol.

For further details about the MSG and PRI parts of a syslog message, see the following sections:

The HEADER message part

The HEADER message part contains a timestamp and the hostname (without the domain name) or the IP address of the device. The timestamp field is the local time in the Mmm dd hh:mm:ss format, where:

  • Mmm is the English abbreviation of the month: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.

  • dd is the day of the month on two digits. If the day of the month is less than 10, the first digit is replaced with a space. (for example, Aug 7.)

  • hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between 00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive.

5.8.1.3 - The MSG message part

This section describes the MSG message part of a syslog message, according to the legacy-syslog or BSD-syslog protocol.

For further details about the HEADER and PRI message parts of a syslog message, see the following sections:

The MSG message part

The MSG part contains the name of the program or process that generated the message, and the text of the message itself. The MSG part is usually in the following format: program[pid]: message text.

5.8.2 - IETF-syslog messages

This section describes the format of a syslog message, according to the IETF-syslog protocol. A syslog message consists of the following parts:

  • HEADER (includes the PRI as well)

  • STRUCTURED-DATA

  • MSG

The following is a sample syslog message (source: https://tools.ietf.org/html/rfc5424):

   <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

The message corresponds to the following format:

   <priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG
  • Facility is 4, severity is 2, so PRI is 34.

  • The VERSION is 1.

  • The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second.

  • The message originated from a host that identifies itself as “mymachine.example.com”.

  • The APP-NAME is “su” and the PROCID is unknown.

  • The MSGID is “ID47”.

  • The MSG is “‘su root’ failed for lonvick…”, encoded in UTF-8.

  • In this example, the encoding is defined by the BOM:

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

  • There is no STRUCTURED-DATA present in the message, this is indicated by “-” in the STRUCTURED-DATA field.

The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.

The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

syslog Message Facilities

Numerical Code

Facility

0

kernel messages

1

user-level messages

2

mail system

3

system daemons

4

security/authorization messages

5

messages generated internally by syslogd

6

line printer subsystem

7

network news subsystem

8

UUCP subsystem

9

clock daemon

10

security/authorization messages

11

FTP daemon

12

NTP subsystem

13

log audit

14

log alert

15

clock daemon

16-23

locally used facilities (local0-local7)

The following table lists the severity values.

Numerical CodeSeverity
0Emergency: system is unusable
1Alert: action must be taken immediately
2Critical: critical conditions
3Error: error conditions
4Warning: warning conditions
5Notice: normal but significant condition
6Informational: informational messages
7Debug: debug-level messages

syslog Message Severities

The HEADER message part

The HEADER part contains the following elements:

  • VERSION: Version number of the syslog protocol standard. Currently this can only be 1.

  • ISOTIMESTAMP: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00.

  • HOSTNAME: The machine that originally sent the message.

  • APPLICATION: The device or application that generated the message

  • PID: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.

  • MESSAGEID: The ID number of the message.

The AxoSyslog application will truncate the following fields:

  • If APP-NAME is longer than 48 characters it will be truncated to 48 characters.

  • If PROC-ID is longer than 128 characters it will be truncated to 128 characters.

  • If MSGID is longer than 32 characters it will be truncated to 32 characters.

  • If HOSTNAME is longer than 255 characters it will be truncated to 255 characters.

The STRUCTURED-DATA message part

The STRUCTURED-DATA message part may contain meta- information about the syslog message, or application-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists of data blocks enclosed in brackets ([]). Every block includes the ID of the block, and one or more name=value pairs. The AxoSyslog application automatically parses the STRUCTURED-DATA part of syslog messages, which can be referenced in macros (for details, see Macros of AxoSyslog). An example STRUCTURED-DATA block looks like:

   [exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]

The MSG message part

The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if the BOM

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

character is present in the message. If the message does not contain the BOM character, the encoding is treated as unknown. Usually messages arriving from legacy sources do not include the BOM character. CRLF characters will not be removed from the message.

5.8.3 - Enterprise-wide message model (EWMM)

The following section describes the structure of log messages using the Enterprise-wide message model or EWMM message format.

The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving AxoSyslog component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first AxoSyslog component in your infrastructure has received it, which is important if you want to forward a message for example, to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.

The following is a sample log message in EWMM format.

   <13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
    {"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
    /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}

The message has the following parts:

  • The header of the complies with the RFC5424 message format, where the PROGRAM field is set to @syslog-ng, and the SDATA field is empty.

  • The MESSAGE part is in JSON format, and contains the actual message, as well as any name-value pairs that AxoSyslog has attached to or extracted from the message. The ${._TAGS} field contains the identifier of the AxoSyslog source that has originally received the message on the first AxoSyslog node.

To send a message in EWMM format, you can use the syslog-ng() destination driver, or the format-ewmm() template function.

To receive a message in EWMM format, you can use the default-destination-drivers() source driver, or the ewmm-parser() parser.

5.9 - Message representation in AxoSylog

When the AxoSyslog application receives a message, it automatically parses the message. The AxoSyslog application can automatically parse log messages that conform to the RFC3164 (BSD or legacy-syslog) or the RFC5424 (IETF-syslog) message formats. If AxoSyslog cannot parse a message, it results in an error.

A parsed syslog message has the following parts:

  • Timestamps

    Two timestamps are associated with every message: one is the timestamp contained within the message (that is, when the sender sent the message), the other is the time when AxoSyslog has actually received the message.

  • Severity

    The severity of the message.

  • Facility

    The facility that sent the message.

  • Tags

    Custom text labels added to the message that are mainly used for filtering. None of the current message transport protocols adds tags to the log messages. Tags can be added to the log message only within AxoSyslog. The AxoSyslog application automatically adds the id of the source as a tag to the incoming messages. Other tags can be added to the message by the pattern database, or using the tags() option of the source.

  • IP address of the sender

    The IP address of the host that sent the message. Note that the IP address of the sender is a hard macro and cannot be modified within AxoSyslog but the associated hostname can be modified, for example, using rewrite rules.

  • Hard macros

    Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. The most important consideration with hard macros is that they are read-only, meaning they cannot be modified using rewrite rules or other means.

  • Soft macros

    Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the AxoSyslog pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within AxoSyslog, for example, using rewrite rules.

Message size and encoding

Internally, AxoSyslog represents every message as UTF-8. The maximal length of the log messages is limited by the log-msg-size() option: if a message is longer than this value, AxoSyslog truncates the message at the location it reaches the log-msg-size() value, and discards the rest of the message.

When encoding is set in a source (using the encoding() option) and the message is longer (in bytes) than log-msg-size() in UTF-8 representation, AxoSyslog splits the message at an undefined location (because the conversion between different encodings is not trivial).

5.10 - Structuring macros, metadata, and other value-pairs

Available in AxoSyslog 3.3 and later.

The AxoSyslog application allows you to select and construct name-value pairs from any information already available about the log message, or extracted from the message itself. You can directly use this structured information, for example, in the following places:

When using value-pairs, there are three ways to specify which information (that is, macros or other name-value pairs) to include in the selection.

  • Select groups of macros using the scope() parameter, and optionally remove certain macros from the group using the exclude() parameter.

  • List specific macros to include using the key() parameter.

  • Define new name-value pairs to include using the pair() parameter.

These parameters are detailed in value-pairs().

5.10.1 - Specifying data types in value-pairs

Prior to version 4.0, AxoSyslog handled every data as strings, and allowed you to convert the strings into other types of data that only certain destinations data formats supported. For example, SQL, MongoDB, JSON, or AMQP support data types like numbers or dates. The AxoSyslog application allows you to specify the data type in templates (this is also called type-hinting or type-casting). If the destination driver supports data types, AxoSyslog converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.

Starting with AxoSyslog 4.0, each name-value pair is a (name, type, value) triplet, and several components of AxoSyslog have typing support, for example, json-parser() and the $(format-json) template function. For details, see the list of supported data types.

Using explicit type-hinting

You can explicitly type-cast a AxoSyslog template to a specific type. To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID"). See the Type-hinting examples and the list of supported data types for details.

Not every destination or other component supports data types. For details, see the list of AxoSyslog components that support data types.

Type-hinting examples

The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.

   mongodb(
        value-pairs(pair("date", datetime("$UNIXTIME"))
            pair("pid", int64("$PID"))
            pair("program", "$PROGRAM"))
            pair("message", "$MESSAGE"))
        )
    );

The following example formats the same fields into JSON.

   $(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)

The following example formats the MESSAGE field as a JSON list.

   $(format-json message=list($MESSAGE))"

Data types in AxoSyslog

The AxoSyslog application currently supports the following data-types.

  • boolean: Converts the data to a boolean value. Anything that begins with a t or 1 is converted to true, anything that begins with an f or 0 is converted to false.
  • datetime: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the $UNIXTIME macro for this purpose.
  • double: A floating-point number.
  • json: A JSON snippet. (Available in AxoSyslog 4.0 and later.)
  • list: The data as a list. For details, see the list manipulation template functions in Template functions of AxoSyslog.
  • null: An unset value.
  • integer: A 32-bit or 64-bit integer, determined by the destination. For example, mongodb uses int32 if the number is less than MAXINT32 and int64 otherwise.
  • string: The data as a string.

Components that support data types

In AxoSyslog 4.0 and later, the following AxoSyslog components that support data types. Other components treat every data as strings.

  • Comparisons in filter expressions: the previously numeric operators are type-aware. The exact comparison depends on the types associated with the values you compare. For details, see Comparing macro values in filters.

  • json-parser() and the $(format-json) template function:

    When using the json-parser(), AxoSyslog converts all elements of the JSON object to name-value pairs. Any type information originally present in the incoming JSON object is retained, and automatically propagated to other AxoSyslog components (for example, a destination) if they support types.

    • Elements without a type are treated as strings.
    • JSON lists (arrays) are converted to AxoSyslog lists, so you can manipulate them using the $(list-*) template functions.
  • set(), groupset() rewrite rules:

    You can set the type of the field. Where you can use of templates in set() and groupset(), you can use type-casting, and the type information is properly promoted. For details, see Specifying data types in value-pairs.

  • db-parser(): The db-parser() rules can associate types with values using the "type" attribute, for example:

    <value name="foobar" type="integer">$PID</value>
    

    The integer is a type-cast that associates $foobar with an integer type. db-parser()’s internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair.

  • add-contextual-data(): Name-value pairs that are populated using add-contextual-data() propagate type information, similarly to db-parser().

  • map-value-pairs(): map-value-pairs() propagates type information.

  • SQL type support: The sql() driver supports types, so that columns with specific types are stored as those types.

  • Template type support: You can cast templates explicitly to a specific type. Templates also propagate type information from macros, template functions, and values in the template string.

  • python() typing: All Python components (sources, destinations, parsers, and template functions) support all data types, except json().

  • On-disk serialized formats (that is, disk buffer): Version 4.0 and newer are compatible with messages serialized with an earlier version, and the format is compatible for downgrades as well. This means that even if a newer version of AxoSyslog serialized a message, older versions and associated tools are able to read it (but drop the type information of course).

5.10.2 - value-pairs()

Type:parameter list of the value-pairs() option
Default:empty string

Description: The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.

Example: Using the value-pairs() option

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message.

   value-pairs(
        scope(nv_pairs core syslog all_macros selected_macros everything)
        exclude("R_*")
        exclude("S_*")
        key(".SDATA.meta.sequenceId")
        pair("MSGHDR" "$PROGRAM[$PID]: ")
    )

The following example selects the same information as the previous example, but converts it into JSON format.

   $(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
        --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
        --pair MSGHDR="$PROGRAM[$PID]: ")

value-pairs() parameters

The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

  1. scope()
  2. exclude()
  3. key()
  4. pair()

exclude()

Type:Space-separated list of macros to remove from the selection created using the scope() option.
Default:empty string

Description: This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the scope() parameter. For example, the following example removes the SDATA macros from the selection.

   value-pairs(
        scope(rfc5424 selected_macros)
        exclude(".SDATA*")
    )

The name of the macro to remove can include wildcards (*, ?). Regular expressions are not supported.

key()

Type:Space-separated list of macros to be included in the selection.
Default:empty string

Description: This option selects the specified macros. The selected macros will be included as MACRONAME = MACROVALUE, that is using key("HOST") will result in HOST = $HOST. You can use wildcards (*, ?) to select multiple macros. For example:

   value-pairs(
        scope(rfc3164)
        key("HOST")
    )
   value-pairs(
        scope(rfc3164)
        key("HOST", "PROGRAM")
    )

omit-empty-values()

Type:flag
Default:N/A

Description: If this option is specified, AxoSyslog does not include value-pairs with empty values in the output. For example: $(format-json --scope none --omit-empty-values) or

   value-pairs(
        scope(rfc3164 selected-macros)
        omit-empty-values()
    )

Available in AxoSyslog version 3.21 and later.

pair()

Type:name-value pairs in "<NAME>" "<VALUE>" format
Default:empty string

Description: This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:

   value-pairs(
        scope(rfc3164)
        pair("TIME" "$HOUR:$MIN")
        pair("MSGHDR" "$PROGRAM[$PID]: ")
    )

rekey()

Type:<pattern-to-select-names>, <list of transformations>
Default:empty string

Description: This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the rekey() option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see glob.

If you want to modify the names of several message fields, see also map-value-pairs: Rename value-pairs to normalize logs.

  • If rekey() is used within a key() option, the name-value pairs specified in the glob of the key() option are transformed.
  • If rekey() is used outside the key() option, every name-value pair of the scope() is transformed.

The following transformations are available:

  • add-prefix("<my-prefix>")

    Adds the specified prefix to every name. For example, rekey( add-prefix("my-prefix."))

  • replace-prefix("<prefix-to-replace>", "<new-prefix>")

    Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, replace-prefix(".class", ".patterndb") changes the beginning tag .class to .patterndb

    This option was called replace() in AxoSyslog version 3.4.

  • lower

    Convert all keys to lowercase. Only supports US ASCII.

  • shift("<number>")

    Cuts the specified number of characters from the beginning of the name.

  • shift-levels("<number>")

    Similar to –shift, but instead of cutting characters, it cuts dot-delimited “levels” in the name (including the initial dot). For example, --shift-levels 2 deletes the prefix up to the second dot in the name of the key: .iptables.SRC becomes SRC

  • upper

    Convert all keys to uppercase. Only supports US ASCII.

Example: Using the rekey() option

The following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters from the names, and adds a new prefix (events.).

   value-pairs(
        key(".cee.*"
            rekey(
                shift(4)
                add-prefix("events.")
            )
        )
    )

The rekey() option can be used with the format-json template-function as well, using the following syntax:

   $(format-json --rekey .cee.* --add-prefix events.)

scope()

Type:Space-separated list of macro groups to be included in the selection.
Default:empty string

Description: This option selects predefined groups of macros. The following groups are available:

  • nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within AxoSyslog and are not originally part of the message, therefore are not included in this group.

  • dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with a dot character are generated within AxoSyslog and are not originally part of the message.

  • all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs and dot-nv-pairs.

  • rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $DATE.

  • rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, $R_DATE, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA..

    The rfc5424 group also has the following alias: syslog-proto. Note that the value of $R_DATE will be listed under the DATE key.

    The rfc5424 group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the $SOURCEIP macro), use the selected-macros group instead.

  • all-macros: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

  • selected-macros: Include the macros of the rfc3164 groups, and the most commonly used metadata about the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.

  • sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA.

  • everything: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

  • none: Reset previously added scopes, for example, to delete automatically-added name-value pairs. The following example deletes every value-pair from the scope, and adds only the ones starting with iptables: $(format-welf --scope none .iptables.*)

For example:

   value-pairs(
        scope(rfc3164 selected-macros)
    )

5.11 - Things to consider when forwarding messages between AxoSyslog hosts

When you send your log messages from a AxoSyslog client through the network to a AxoSyslog server, you can use different protocols and options. Every combination has its advantages and disadvantages. The most important thing is to use matching protocols and options, so the server handles the incoming log messages properly.

In AxoSyslog you can change many aspects of the network communication. First of all, there is the structure of the messages itself. Currently, AxoSyslog supports two standard syslog protocols: the BSD (RFC3164) and the syslog (RFC5424) message format.

These RFCs describe the format and the structure of the log message, and add a (lightweight) framing around the messages. You can set this framing/structure by selecting the appropriate driver in AxoSyslog. There are two drivers you can use: the network() driver and the syslog() driver. The syslog() driver is for the syslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.

The tcp() and udp() drivers are now deprecated, they are essentially equivalent with the network(transport(tcp)) and network(transport(udp)) drivers.

In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCP and UDP, and optionally also higher-level transport protocols: TLS (over TCP. To complicate things a bit more, you can configure the network() driver (corresponding to the BSD (RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used in RFC5424) using the flag(syslog-protocol) option.

Because some combination of drivers and options are invalid, you can use the following drivers and options as sources and as destinations:

  1. syslog(transport(tcp))

  2. syslog(transport(udp))

  3. syslog(transport(rltp))

  4. syslog(transport(tls))

  5. syslog(transport(rltp(tls-required(yes)))

  6. network(transport(tcp))

  7. network(transport(udp))

  8. network(transport(rltp))

  9. network(transport(tls))

  10. network(transport(rltp(tls-required(yes)))

  11. network(transport(tcp) flag(syslog-protocol))

  12. network(transport(udp) flag(syslog-protocol))

  13. network(transport(rltp)flag(syslog-protocol))

  14. network(transport(tls) flag(syslog-protocol))

  15. network(transport(rltp(tls-required(yes)) flag(syslog-protocol))

If you use the same driver and options in the destination of your AxoSyslog client and the source of your AxoSyslog server, everything should work as expected. Unfortunately there are some other combinations, that seem to work, but result in losing parts of the messages. The following table show the combinations:

Source \ Destinationsyslog/tcpsyslog/udpsyslog/tlsnetwork/tcpnetwork/udpnetwork/tlsnetwork/tcp/flagnetwork/udp/flagnetwork/tls/flag
syslog/tcp--!--!--
syslog/udp---!--!-
syslog/tls----!--!
network/tcp-----✔?--
network/udp-✔?----✔?-
network/tls-------✔?
network/tcp/flag!--!----
network/udp/flag-!--!---
network/tls/flag--!--!--

Source-destination driver combinations

  • - This method does not work. The logs will not get to the server.

  • This method works.

  • ! This method has some visible drawbacks. The logs go through, but some of the values are missing/misplaced/and so on.

  • ✔? This method seems to work, but it is not recommended because this can change in a future release.

6 - The configuration file

6.1 - Location of the configuration file

To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.

  • Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
  • Containers: When running AxoSyslog in a container, typically you map an external file to the /etc/syslog-ng/syslog-ng.conf file within the container. Check the mapped volumes of your container, and edit the external file.
  • Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog collector Helm chart.

6.2 - The configuration syntax in detail

Every syslog-ng.conf configuration file must begin with a line containing version information. For syslog-ng version 4.5, this line looks like:

   @version: 4.5.0
  • If the configuration file does not contain the version information, syslog-ng assumes that the file is for version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later can operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.
  • @version: current sets the configuration version to the currently installed version.

Example: A simple configuration file

The following is a very simple configuration file for syslog-ng: it collects the internal messages of syslog-ng and the messages from /dev/log into the /var/log/messages_syslog-ng.log file.

   @version: 4.5.0
    source s_local {
        unix-dgram("/dev/log"); internal();
    };
    destination d_file {
        file("/var/log/messages_syslog-ng.log");
    };
    log {
        source(s_local); destination(d_file);
    };

As a syslog-ng user described on a mailing list:

Alan McKinnon

The syslog-ng’s config file format was written by programmers for programmers to be understood by programmers. That may not have been the stated intent, but it is how things turned out. The syntax is exactly that of C, all the way down to braces and statement terminators.

  • The main body of the configuration file consists of object definitions: sources, destinations, logpaths define which log message are received and where they are sent. All identifiers, option names and attributes, and any other strings used in the syslog-ng.conf configuration file are case sensitive. Object definitions (also called statements) have the following syntax:

        type-of-the-object identifier-of-the-object {<parameters>};
    
    • Type of the object: One of source, destination, log, filter, parser, rewrite rule, or template.

    • Identifier of the object: A unique name identifying the object. When using a reserved word as an identifier, enclose the identifier in quotation marks.

      All identifiers, attributes, and any other strings used in the syslog-ng.conf configuration file are case sensitive.

    • Parameters: The parameters of the object, enclosed in braces {parameters}.

    • Semicolon: Object definitions end with a semicolon (;).

    For example, the following line defines a source and calls it s_internal.

        source s_internal {
            internal();
        };
    

    The object can be later referenced in other statements using its ID, for example, the previous source is used as a parameter of the following log statement:

        log {
            source(s_internal); destination(d_file);
        };
    
  • The parameters and options within a statement are similar to function calls of the C programming language: the name of the option followed by a list of its parameters enclosed within brackets and terminated with a semicolon.

        option(parameter1, parameter2); option2(parameter1, parameter2);
    

    For example, the file() driver in the following source statement has three options: the filename (/var/log/apache/access.log), follow-freq(), and flags(). The follow-freq() option also has a parameter, while the flags() option has two parameters.

        source s_tail {
            file("/var/log/apache/access.log" follow-freq(1) flags(no-parse, validate-utf8));
        };
    

    Objects may have required and optional parameters. Required parameters are positional, meaning that they must be specified in a defined order. Optional parameters can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object.

    Example: Using required and optional parameters

    The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect:

        source s_demo_stream1 {
            unix-stream("<path-to-socket>" max-connections(10) group(log));
        };
        source s_demo_stream2 {
            unix-stream("<path-to-socket>" group(log) max-connections(10));
        };
    
    • Some options are global options, or can be set globally, for example, whether AxoSyslog should use DNS resolution to resolve IP addresses. Global options are detailed in Global options.

          options {
              use-dns(no);
          };
      
    • Objects can be used before definition.

    • Objects can be defined inline as well. This is useful if you use the object only once (for example, a filter). For details, see Defining configuration objects inline.

    • To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

          # Comment: This is a stream source
          source s_demo_stream {
              unix-stream("<path-to-socket>" max-connections(10) group(log));
          };
      

6.3 - Notes about the configuration syntax

When you are editing the syslog-ng.conf configuration file, note the following points:

  • The configuration file can contain a maximum of 6665 source / destination / log elements.

  • When writing the names of options and parameters (or other reserved words), the hyphen (-) and underscore (_) characters are equivalent, for example, max-connections(10) and max_connections(10) are both correct.

  • Numbers can be prefixed with + or - to indicate positive or negative values. Numbers beginning with zero (0) or 0x are treated as octal or hexadecimal numbers, respectively.

    Starting with AxoSyslog version 3.5, you can use suffixes for kilo-, mega-, and gigabytes. Use the Kb, Mb, or Gb suffixes for the base-10 version, and Kib, Mib, or Gib for the base-2 version. That is, 2MB means 2000000, while 2MiB means 2097152. For example, to set the log-msg-size() option to 2000000 bytes, use log-msg-size(2Mb).

  • You can use commas (,) to separate options or other parameters for readability, AxoSyslog completely ignores them. The following declarations are equivalent:

        source s_demo_stream {
            unix-stream("<path-to-socket>" max-connections(10) group(log));
        };
        source s_demo_stream {
            unix-stream("<path-to-socket>", max-connections(10), group(log));
        };
    
  • When enclosing object IDs (for example, the name of a destination) between double-quotes ("mydestination"), the ID can include whitespace as well, for example:

        source "s demo stream" {
            unix-stream("<path-to-socket>" max-connections(10) group(log));
        };
    
  • For notes on using regular expressions, see Regular expressions.

  • You can use if {}, elif {}, and else {} blocks to configure conditional expressions. For details, see if-else-elif: Conditional expressions.

6.4 - Defining configuration objects inline

Starting with AxoSyslog 3.4, you can define configuration objects inline, where they are actually used, without having to define them in a separate placement. This is useful if you need an object only once, for example, a filter or a rewrite rule. Every object can be defined inline: sources, destinations, filters, parsers, rewrite rules, and so on.

To define an object inline, use braces instead of parentheses. That is, instead of <object-type> (<object-id>);, you use <object-type> {<object-definition>};

Example: Using inline definitions

The following two configuration examples are equivalent. The first one uses traditional statements, while the second uses inline definitions.

   source s_local {
        system();
        internal();
    };
    destination d_local {
        file("/var/log/messages");
    };
    log {
        source(s_local);
        destination(d_local);
    };
   log {
        source {
            system();
            internal();
        };
        destination {
            file("/var/log/messages");
        };
    };

6.5 - Using channels in configuration objects

Starting with AxoSyslog 3.4, every configuration object is a log expression. Every configuration object is essentially a configuration block, and can include multiple objects. To reference the block, only the top-level object must be referenced. That way you can use embedded log statements, junctions and in-line object definitions within source, destination, filter, rewrite and parser definitions. For example, a source can include a rewrite rule to modify the messages received by the source, and that combination can be used as a simple source in a log statement. This feature allows you to preprocess the log messages very close to the source itself.

To embed multiple objects into a configuration object, use the following syntax. Note that you must enclose the configuration block between braces instead of parenthesis.

   <type-of-top-level-object> <name-of-top-level-object> {
        channel {
            <configuration-objects>
        };
    };

Example: Using channels

For example, to process a log file in a specific way, you can define the required processing rules (parsers and rewrite expressions) and combine them in a single object:

   source s_apache {
        channel {
            source {
                file("/var/log/apache/error.log");
            };
            parser(p_apache_parser);
        };
    };
    log {
        source(s_apache); ...
    };

The s_apache source uses a file source (the error log of an Apache webserver) and references a specific parser to process the messages of the error log. The log statement references only the s_apache source, and any other object in the log statement can already use the results of the p_apache_parserparser.

6.6 - Global and environmental variables

You can define global variables in the configuration file. Global variables are actually name-value pairs. When syslog-ng processes the configuration file during startup, it automatically replaces name with value. To define a global variable, use the following syntax:

   @define name "value"

The value can be any string, but special characters must be escaped (for details, see Regular expressions). To use the variable, insert the name of the variable enclosed between backticks (`, similarly to using variables in Linux or UNIX shells) anywhere in the configuration file. If backticks are meant literally, repeat the backticks to escape them. For example:

``not-substituted-value``

The value of the global variable can be also specified using the following methods:

  • Without any quotes, as long as the value does not contain any spaces or special characters. In other words, it contains only the following characters: a-zA-Z0-9_..

  • Between apostrophes, in case the value does not contain apostrophes.

  • Between double quotes, in which case special characters must be escaped using backslashes (\\).

Example: Using global variables

For example, if an application is creating multiple log files in a directory, you can store the path in a global variable, and use it in your source definitions.

   @define mypath "/opt/myapp/logs"
    source s_myapp_1 {
        file("`mypath`/access.log" follow-freq(1));
    };
    source s_myapp_2 {
        file("`mypath`/error.log" follow-freq(1));
    };
    source s_myapp_3 {
        file("`mypath`/debug.log" follow-freq(1));
    };

The AxoSyslog application will interpret this as:

   @define mypath "/opt/myapp/logs"
    source s_myapp_1 {
        file("/opt/myapp/logs/access.log" follow-freq(1));
    };
    source s_myapp_2 {
        file("/opt/myapp/logs/error.log" follow-freq(1));
    };
    source s_myapp_3 {
        file("/opt/myapp/logs/debug.log" follow-freq(1));
    };

6.7 - Configuration identifier

Starting with AxoSyslog version 4.2, you can specify a configuration identifier in the syslog-ng.conf file, for example:

@config-id: cfg-20230404-13-g02b0850fc

This can be useful in managed environments, where AxoSyslog instances and their configuration are automatically deployed or generated.

To show the configuration ID, run syslog-ng-ctl config --id

This returns the ID of the currently active configuration, and the SHA256 hash of the configuration (the hash of the output of the syslog-ng-ctl config --preprocessed command). The output is similar to:

cfg-20230404-13-g02b0850fc (08ddecfa52a3443b29d5d5aa3e5114e48dd465e195598062da9f5fc5a45d8a83)

6.8 - Using modules

To increase its flexibility and simplify the development of additional modules, the AxoSyslog application is modular. The majority of AxoSyslog’s functionality is in separate modules. As a result, it is also possible to fine-tune the resource requirements of AxoSyslog (for example, by loading only the modules that are actually used in the configuration, or simply omitting modules that are not used but require large amount of memory).

Each module contains one or more plugins that add some functionality to AxoSyslog (for example, a destination or a source driver).

  • To display the list of available modules, run the syslog-ng --version command.

  • To display the description of the available modules, run the syslog-ng --module-registry command.

  • To customize which modules AxoSyslog automatically loads when AxoSyslog starts, use the --default-modules command-line option of AxoSyslog.

  • To request loading a module from the AxoSyslog configuration file, see Loading modules.

For details on the command-line parameters of AxoSyslog mentioned in the previous list, see the AxoSyslog man page at The syslog-ng manual page.

6.8.1 - Loading modules

The AxoSyslog application loads every available module during startup.

To load a module that is not loaded automatically, include the following statement in the AxoSyslog configuration file:

   @module <module-name>

Note the following points about the @module statement:

  • The @module statement is a top-level statement, that is, it cannot be nested into any other statement. It is usually used immediately after the @version statement.

  • Every @module statement loads a single module: loading multiple modules requires a separate @module statement for every module.

  • In the configuration file, the @module statement of a module must be earlier than the module is used.

Use the @requires statement to ensure that the specified module is loaded

To ensure that a module is loaded, include the following statement in the AxoSyslog configuration file or the external files included in the configuration file:

   @requires <module-name>

6.8.2 - Listing configuration options

Starting with AxoSyslog 3.25, you can use the syslog-ng-cfg-db.py utility to list the available options of configuration objects. For example, you can list all the options that can be set in the file source, and so on.

The syslog-ng-cfg-db.py utility has the following options:

  • The following command lists the contexts that the utility supports.

        syslog-ng-cfg-db.py
    
  • The following command lists the available drivers of a context:

        syslog-ng-cfg-db.py -c <source|destination>
    
  • The following command lists the available options of a specific driver and specifies the context and the driver:

        syslog-ng-cfg-db.py -c <source|destination> -d <driver>
    

    For example, to list the options of the kafka-c() destination driver:

        syslog-ng-cfg-db.py -c destination -d kafka-c
    

    The output includes the available options of the driver in alphabetical order, and the type of the option. For example:

        destination kafka-c(
            bootstrap-servers/kafka-bootstrap-servers(<string>)
            client-lib-dir(<string>)
            config/option()
            config/option(<string> <arrow> <string-or-number>)
            config/option(<string> <string-or-number>)
            flush-timeout-on-reload(<number>)
            flush-timeout-on-shutdown(<number>)
            frac-digits(<number>)
            key(<string>)
            local-time-zone/time-zone(<string>)
            log-fifo-size(<number>)
            message/template(<string>)
            on-error(<string>)
            persist-name(<string>)
            poll-timeout(<number>)
            properties-file(<path>)
            send-time-zone(<string>)
            sync-send(<yesno>)
            throttle(<number>)
            time-zone(<string>)
            topic(<string>)
            ts-format(<string>)
            workers(<number>)
            config/option(
                <string>(<string-or-number>)
            )
            key(
                <identifier>(<string>)
            )
            message/template(
                <identifier>(<string>)
            )
        )
    

6.8.3 - Visualize the configuration

Starting with AxoSyslog 3.25, you can visualize the configuration of a running AxoSyslog instance using the syslog-ng-ctl --export-config-graph command. The command walks through the effective configuration, and exports it as a graph into a JSON structure.

The resulting JSON file can be converted into DOT file format that visualization tools (for example, Graphviz) can use. The package includes a Python script to convert the exported JSON file into DOT format: <syslog-ng-installation-directory>/contrib/scripts/config-graph-json-to-dot.py

You can convert the DOT file into PNG or PDF format using external tools.

6.9 - Managing complex configurations

The following sections describe some methods that can be useful to simplify the management of large-scale AxoSyslog installations.

6.9.1 - Including configuration files

The AxoSyslog application supports including external files in its configuration file, so parts of its configuration can be managed separately. To include the contents of a file in the AxoSyslog configuration, use the following syntax:

   @include "<filename>"

This imports the entire file into the configuration of AxoSyslog, at the location of the include statement. The <filename> can be one of the following:

  • A filename, optionally with full path. The filename (not the path) can include UNIX-style wildcard characters (*, ?). When using wildcard characters, AxoSyslog will include every matching file. For details on using wildcard characters, see Types and options of regular expressions.

  • A directory. When including a directory, AxoSyslog will try to include every file from the directory, except files beginning with a ~ (tilde) or a . (dot) character. Including a directory is not recursive. The files are included in alphabetic order, first files beginning with uppercase characters, then files beginning with lowercase characters. For example, if the directory contains the a.conf, B. conf, c.conf, D.conf files, they will be included in the following order: B.conf, D. conf, a.conf, c.conf.

When including configuration files, consider the following points:

  • The default path where AxoSyslog looks for the file depends on where AxoSyslog is installed. The syslog-ng --version command displays this path as Include-Path.

  • Defining an object twice is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file. If an object is defined twice (for example, the original configuration file and the file imported into this configuration file both define the same option, source, or other object), then the object that is defined later in the configuration file will be effective. For example, if you set a global option at the beginning of the configuration file, and later include a file that defines the same option with a different value, then the option defined in the imported file will be used.

    • Files can be embedded into each other: the included files can contain include statements as well, up to a maximum depth of 15 levels.

    • You cannot include complete configuration files into each other, only configuration snippets can be included. This means that the included file cannot have a @version statement.

    • Include statements can only be used at top level of the configuration file. For example, the following is correct:

          @version: 4.5.0
          @include "example.conf"
      

      But the following is not:

          source s_example {
              @include "example.conf"
          };
      

6.9.2 - Reusing configuration blocks

To create a reusable configuration snippet and reuse parts of a configuration file, you have to define the block (for example, a source) once, and reference it later. Any AxoSyslog object can be a block. Use the following syntax to define a block:

   block type name() {<contents of the block>};

Type must be one of the following: destination, filter, log, options, parser, rewrite, root, source. The root blocks can be used in the “root” context of the configuration file, that is, outside any other statements.

Note that options can be used in blocks only in version 3.22 and later.

Blocks may be nested into each other, so for example, a block can be built from other blocks. Blocks are somewhat similar to C++ templates.

The type and name combination of each block must be unique, that is, two blocks can have the same name if their type is different.

To use a block in your configuration file, you have to do two things:

  • Include the file defining the block in the syslog-ng.conf file — or a file already included into syslog-ng.conf. Version 3.7 and newer automatically includes the *.conf files from the <directory-where-syslog-ng-is-installed>/scl/*/ directories.

  • Reference the name of the block in your configuration file. This will insert the block into your configuration. For example, to use a block called myblock, include the following line in your configuration:

        myblock()
    

    Blocks may have parameters, but even if they do not, the reference must include opening and closing parentheses like in the previous example.

The contents of the block will be inserted into the configuration when AxoSyslog is started or reloaded.

Example: Reusing configuration blocks

Suppose you are running an application on your hosts that logs into the /opt/var/myapplication.log file. Create a file (for example, myblocks.conf) that stores a source describing this file and how it should be read:

   block source myappsource() {
            file("/opt/var/myapplication.log" follow-freq(1) default-facility(syslog)); };

Include this file in your main syslog-ng.conf configuration file, reference the block, and use it in a logpath:

   @version: 4.5.0
    @include "<correct/path>/myblocks.conf"
    source s_myappsource { myappsource(); };
    ...
    log { source(s_myappsource); destination(...); };

To define a block that defines more than one object, use root as the type of the block, and reference the block from the main part of the AxoSyslog configuration file.

Example: Defining blocks with multiple elements

The following example defines a source, a destination, and a log path to connect them.

   block root mylogs() {
        source s_file {
            file("/var/log/mylogs.log" follow-freq(1));
        };
        destination d_local {
            file("/var/log/messages");
        };
        log {
            source(s_file); destination(d_local);
        };
    };

Mandatory parameters

You can express in block definitons that a parameter is mandatory by defining it with empty brackets (). In this case, the parameter must be overridden in the reference block. Failing to do so will result in an error message and initialization failure.

To make a parameter expand into nothing (for example, because it has no default value, like hook-commands() or tls()), insert a pair of double quote marks inside the empty brackets: ("")

Example: Mandatory parameters

The following example defines a TCP source that can receive the following parameters: the port where AxoSyslog listens (localport), and optionally source flags (flags).

   block source my_tcp_source(localport() flags("")) {
        network(port(`localport`) transport(tcp) flags(`flags`));
    };

Because localport is defined with empty brackets (), it is a mandatory parameter. However, the flags parameter is not mandatory, because it is defined with an empty double quote bracket pair (""). If you do not enter a specific value when referencing this parameter, the value will be an empty string. This means that in this case

   my_tcp_source(localport(8080))

will be expanded to:

   network(port(8080) transport(tcp) flags());

Passing arguments to configuration blocks

Configuration blocks can receive arguments as well. The parameters the block can receive must be specified when the block is defined, using the following syntax:

   block type block_name(argument1(<default-value-of-the-argument>) argument2(<default-value-of-the-argument>) argument3())

If an argument does not have a default value, use an empty double quote bracket pair ("") after the name of the argument. To refer the value of the argument in the block, use the name of the argument between backticks, for example:

`argument1`

Example: Passing arguments to blocks

The following sample defines a file source block, which can receive the name of the file as a parameter. If no parameter is set, it reads messages from the /var/log/messages file.

   block source s_logfile (filename("messages")) {
        file("/var/log/`filename`" );
    };
    
    source s_example {
        s_logfile(filename("logfile.log"));
    };

If you reference the block with more arguments then specified in its definition, you can use these additional arguments as a single argument-list within the block. That way, you can use a variable number of optional arguments in your block. This can be useful when passing arguments to a template, or optional arguments to an underlying driver.

The three dots () at the end of the argument list refer to any additional parameters. It tells AxoSyslog that this macro accepts __VARARGS__, therefore any name-value pair can be passed without validation. To reference this argument-list, insert __VARARGS__ to the place in the block where you want to insert the argument-list. Note that you can use this only once in a block.

The following definition extends the logfile block from the previous example, and passes the optional arguments (follow-freq(1) flags(no-parse)) to the file() source.

   block source s_logfile(filename("messages") ...) {
        file("/var/log/`filename`" `__VARARGS__`);
    };
    
    source s_example {
        s_logfile(
            filename("logfile.log")
            follow-freq(1)
            flags(no-parse)
        );
    };

Example: Using arguments in blocks

The following example is the code of the pacct() source driver, which is actually a block that can optionally receive two arguments.

   block source pacct(file("/var/log/account/pacct") follow-freq(1) ...) {
        file("`file`" follow-freq(`follow-freq`) format("pacct") tags(".pacct") `__VARARGS__`);
    };

Example: Defining global options in blocks

The following example defines a block called setup-dns() to set DNS-related settings at a single place.

   block options setup-dns(use-dns()) {
        keep-hostname(no);
        use-dns(`use-dns`);
        use-fqdn(`use-dns`);
    dns-cache(`use-dns`);
    };
    
    options {
        setup-dns(use-dns(yes));
    };

6.9.3 - Generating configuration blocks from a script

Purpose:

The AxoSyslog application can automatically execute scripts when it is started, and can include the output of such script in the configuration file. To create and use a script that generates a part of the AxoSyslog configuration file (actually, a configuration block), complete the following steps. The steps include examples for collecting Apache access log files (access.log) from subdirectories, but you can create any script that creates a valid AxoSyslog configuration snippet.

Steps:

  1. Navigate to the directory where you have installed AxoSyslog (for example, /opt/syslog-ng/share/include/scl/), and create a new directory, for example, apache-access-logs. The name of the directory will be used in the AxoSyslog configuration file as well, so use a descriptive name.

  2. Create a file called plugin.conf in this new directory.

  3. Edit the plugin.conf file and add the following line:

        @module confgen context(source) name(<directory-name>) exec("`scl-root`/<directory-name>/<my-script>")
    

    Replace <directory-name> with the name of the directory (for example, apache-access-logs), and <my-script> with the filename of your script (for example, apache-access-logs.sh). You can reference the script in your AxoSyslog configuration file as a configuration block using the value name option.

    The context option determines the type of the configuration snippet that the script generates, and must be one of the following: destination, filter, log, parser, rewrite, root, source. The root blocks can be used in the “root” context of the configuration file, that is, outside any other statements. In the example, context(source) means that the output of the script will be used within a source statement.

    You can pass parameters to the script. In the script these parameters are available as environment variables, and have the confgen_ prefix. For example, passing the --myparameter parameter becomes available in the script as the confgen_myparameter environment variable.

  4. Write a script that generates the output you need, and formats it to a configuration snippet that AxoSyslog can use. The filename of the script must match with the filename used in plugin.conf, for example, apache-access-logs.sh.

    The following example checks the /var/log/apache2/ directory and its subdirectories, and creates a source driver for every directory that contains an access.log file.

        #!/bin/bash
        for i in `find /var/log/apache2/ -type d`; do
            echo "file(\"$i/access.log\" flags(no-parse) program-override(\"apache2\"));";
        done;
    

    The script generates an output similar to this one, where service* is the actual name of a subdirectory:

        file("/var/log/apache2/service1/access.log" flags(no-parse) program-override("apache2"));
        file("/var/log/apache2/service2/access.log" flags(no-parse) program-override("apache2"));
    
  5. Include the plugin.conf file in the syslog-ng.conf file — or a file already included into syslog-ng.conf. Version 3.7 and newer automatically includes the *.conf files from the <directory-where-syslog-ng-is-installed>/scl/*/ directories. For details on including configuration files, see Including configuration files.

  6. Add the block you defined in the plugin.conf file to your AxoSyslog configuration file. You can reference the block using the value of the name option from the plugin.conf file, followed by parentheses, for example, apache-access-logs(). Make sure to use the block in the appropriate context of the configuration file, for example, within a source statement if the value of the context option in the plugin.conf file is source.

        @include "scl.conf"
        ...
        source s_apache {
            file("/var/log/apache2/access.log" flags(no-parse) program-override("apache2"));
            file("/var/log/apache2/error.log" flags(no-parse) program-override("apache2"));
            file("/var/log/apache2/ssl.log" flags(no-parse) program-override("apache2"));
            apache-access-logs();
        };
    
        log {
            source(s_apache); destination(d_central);
        };
        ...
    
  7. Check if your modified AxoSyslog configuration file is syntactically correct using the syslog-ng --syntax-only command.

  8. If your modified configuration is syntactically correct, load the new configuration file using the syslog-ng-ctl reload command.

6.10 - Python code in external files

You can extend and customize AxoSyslog easily by writing destinations, parsers, template functions, and sources in Python.

Instead of writing Python code into your AxoSyslog configuration file, you can store the Python code for your Python object in an external file. That way, it is easier to write, maintain, and debug the code. You can store the Python code in any directory in your system, but make sure to include it in your Python path.

When referencing a Python class from an external file in the class() option of a Python block in the AxoSyslog configuration file, the class name must include the name of the Python file containing the class, without the path and the .py extension. For example, if the MyDestination class is available in the /etc/syslog-ng/etc/pythonexample.py file, use class("pythonexample.MyDestination"):

   destination d_python_to_file {
        python(
            class("pythonexample.MyDestination")
        );
    };
    log {
        source(src);
        destination(d_python_to_file);
    };

If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.

In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".

To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.

6.11 - Logging from your Python code

You can extend and customize AxoSyslog easily by writing destinations, parsers, template functions, and sources in Python.

To debug and troubleshoot your Python code, AxoSyslog allows you to use the logger() method to send log messages to the internal() source of AxoSyslog. That way the diagnostic messages of your Python code are treated the same way as other such log messages of AxoSyslog. This has the following benefits:

  • The logger() method respects the log level settings of AxoSyslog. You can write error, warning, info, debug, and trace level messages.

  • You can follow what your Python code is doing even if AxoSyslog is running as a daemon in the background.

Logging to the internal() source is available in AxoSyslog version 3.20 and later.

To send log messages to the internal() source from Python

  1. Add the following import to your Python code:

        import syslogng
    
  2. Create a logger object:

        logger = syslogng.Logger()
    
  3. Use the logger object in your Python code, for example:

        logger.info("This is a sample log message send from the Python code.")
    

    You can use the following log levels: logger.error, logger.warning, logger.info, logger.debug, logger.trace

  4. Make sure that your AxoSyslog configuration includes the internal() source, for example:

        source s_internal { internal(); };
        destination d_internal { file("/var/log/internal.txt"); };
        log {source(s_internal); destination(d_internal); };
    

7 - source: Read, receive, and collect log messages

7.1 - How sources work

A source is where AxoSyslog receives log messages. Sources consist of one or more drivers, each defining where and how messages are received.

To define a source, add a source statement to the syslog-ng.conf configuration file using the following syntax:

   source <identifier> {
        source-driver(params); source-driver(params); ...
    };

Example: A simple source statement

The following source statement receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address.

   source s_demo_tcp {
        network(ip(10.1.2.3) port(1999));
    };

Example: A source statement using two source drivers

The following source statement receives messages on the 1999 TCP port and the 1999 UDP port of the interface having the 10.1.2.3 IP address.

   source s_demo_two_drivers {
        network(ip(10.1.2.3) port(1999));
        network(ip(10.1.2.3) port(1999) transport("udp"));
    };

Example: Setting default priority and facility

If the message received by the source does not have a proper syslog header, you can use the default-facility() and default-priority() options to set the facility and priority of the messages. Note that these values are applied only to messages that do not set these parameters in their header.

   source headerless_messages { network(default-facility(syslog) default-priority(emerg)); };

Define a source only once. The same source can be used in several log paths. Duplicating sources causes AxoSyslog to open the source (TCP/IP port, file, and so on) more than once, which might cause problems. For example, include the /dev/log file source only in one source statement, and use this statement in more than one log path if needed.

To collect log messages on a specific platform, it is important to know how the native syslogd communicates on that platform. The following table summarizes the operation methods of syslogd on some of the tested platforms:

Communication methods used between the applications and syslogd

PlatformMethod
LinuxA SOCK_DGRAM unix socket named /dev/log. Newer distributions that use systemd collect log messages into a journal file.
BSD flavorsA SOCK_DGRAM unix socket named /var/run/log.
Solaris (2.5 or below)An SVR4 style STREAMS device named /dev/log.
Solaris (2.6 or above)In addition to the STREAMS device used in earlier versions, 2.6 uses a new multithreaded IPC method called door. By default the door used by syslogd is /etc/.syslog_door.
HP-UX 11 or laterHP-UX uses a named pipe called /dev/log that is padded to 2048 bytes, for example, source s_hp-ux {pipe ("/dev/log" pad-size(2048)}.
AIX 5.2 and 5.3A SOCK_STREAM or SOCK_DGRAM unix socket called /dev/log.

Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using the SOCK_STREAM style — as used under Linux — is called unix-stream.

Example: Source statement on a Linux based operating system

The following source statement collects the following log messages:

  • internal(): Messages generated by syslog-ng.

  • network(transport(“udp”)): Messages arriving to the 514/UDP port of any interface of the host.

  • unix-dgram("/dev/log");: Messages arriving to the /dev/log socket.

   source s_demo {
        internal();
        network(transport("udp"));
        unix-dgram("/dev/log");
    };

Sources list

The following table lists the source drivers available in syslog-ng.

Source drivers available in syslog-ng

NameDescription
file()Opens the specified file and reads messages.
internal()Messages generated internally in syslog-ng.
kubernetes()Collects container logs managed by the Kubelet.
mbox()Read email messages from local mbox files, and convert them to multiline log messages.
network()Receives messages from remote hosts using the BSD-syslog protocol over IPv4 and IPv6. Supports the TCP, UDP, and TLS network protocols.
nodejs()Receives JSON messages from nodejs applications.
osquery()Run osquery queries, and convert their results into log messages.
pacct()Reads messages from the process accounting logs on Linux.
pipe()Opens the specified named pipe and reads messages.
program()Opens the specified application and reads messages from its standard output.
python() and python-fetcher()Receive or fetch messages using a custom source written in Python.
snmptrap()Read and parse the SNMP traps of the Net-SNMP’s snmptrapd application.
sun-stream(), sun-streams()Opens the specified STREAMS device on Solaris systems and reads incoming messages.
syslog()Listens for incoming messages using the new IETF-standard syslog protocol.
system()Automatically detects which platform AxoSyslog is running on, and collects the native log messages of that platform.
systemd-journal()Collects messages directly from the journal of platforms that use systemd.
systemd-syslog()Collects messages from the journal using a socket on platforms that use systemd.
unix-dgram()Opens the specified unix socket in SOCK_DGRAM mode and listens for incoming messages.
unix-stream()Opens the specified unix socket in SOCK_STREAM mode and listens for incoming messages.
stdin()Collects messages from the standard input stream.
wildcard-file()Reads messages from multiple files and directories.

7.2 - Arr logs

Starting with version 4.7.0, AxoSyslog can collect logs of the Lidarr, Prowlarr, Radarr, Readarr, and Sonarr (often referred to as “*Arr” or “*Arrs”) applications.

Use the new *arr() sources to read various *arr logs:

  • lidarr()
  • prowlarr()
  • radarr()
  • readarr()
  • sonarr()
  • whisparr()

Example minimal config:

source s_radarr {
    radarr(
    dir("/path/to/my/radarr/log/dir")
    );
};

The logging module is stored in the <prefix><module> name-value pair, for example: .radarr.module => ImportListSyncService.

You can modify the prefix with the prefix() option.

This driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

7.3 - Collect native macOS system logs

Starting with version 4.6.0, AxoSyslog can collect logs on macOS using its native OSLog framework using the darwin-oslog() and darwin-oslog-stream() source drivers.

darwin-oslog()

This source is based on the native OSLog Framework to read logs from the local store of the unified logging system on darwin OSes. The AxoSyslog system() source automatically uses this new source on darwin platforms if the darwinosl plugin is available. This plugin is available only on macOS 10.15 Catalina and above, the first version that has the OSLog API.

Note that the persistent OSLog store usually keeps about 7 days of logs on disk.

The darwin-oslog() source has the following options:

filter-predicate()

Type:string
Default:(eventType == 'logEvent' || eventType == 'lossEvent' || eventType == 'stateEvent' || eventType == 'userActionEvent') && (logType != 'debug')

Description: String for native macOS log message filtering using predicates. For example, the following predicate selects AirDrop logs: subsystem=="com.apple.sharing" and category=="AirDrop"

do-not-use-bookmark()

Type:boolean
Default:no

Description: By default, AxoSyslog continues to read the logs from the last remembered position after a restart. If this option is set to yes, it will always start reading from the end or beginning of the available log list (depending on the setting of the go-reverse() option).

fetch-delay()

Type:integer
Default:10000

Description: Controls the time AxoSyslog waits between reading and sending log messages. This is a fraction of a second, where wait_time = 1 second / n, so n=1 means that only about 1 log is sent in each second, and n=1000000 means only 1 microsecond is the delay between read/write attempts. The maximal value of this parameter is 1000000. Note that increasing the value of this parameter (thus lowering delay time) can increase log feed performance, but at the same time could increase system load.

fetch-retry-delay()

Type:integer
Default:1

Description: Controls how many seconds AxoSyslog waits before trying to check for new logs if there were no more logs to read the last time.

go-reverse()

Type:boolean
Default:no

Description: Set it to yes to process the logs in a reverse order (from latest to oldest).

log-fetch-limit()

Type:integer
Default:0 (no limit)

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

max-bookmark-distance()

Type:integer
Default:0 (no limit) [seconds]

Description: The maximum distance in seconds that a bookmark can point backwards. That is, if AxoSyslog was stopped for 10 minutes and max-bookmark-distance() is set to 60, then AxoSyslog will start reading the logs from 60 seconds before the startup, losing 9 minutes of logs.

read-old-records()

Type:boolean
Default:no

Description: If set to yes, AxoSyslog starts reading logs from the oldest available log when it’s first started on a system, or if there are no bookmarks for some reason

darwin-oslog-stream()

This source is a wrapper around the OS command line log stream command that provides a live log stream feed. Unlike the darwin-oslog() source, the live stream can contain non-persistent log events as well. This might result in a large number of log events every second.

This source has only one option.

params()

Type:string
Default:--type log --type trace --level info --level debug

Description: A string that can contain all the possible params the macOS log tool can accept. The source uses the “–style” internally (defaults to ndjson), so use templates or rewrite rules to format the final output. Use the def-osl-stream-params string to reference the default values when extending them with your own.

For a full reference, see the output of the log --help stream and man log commands.

7.4 - default-network-drivers: Receive and parse common syslog messages

The default-network-drivers() source is a special source that uses multiple source drivers to receive and parse several different types of syslog messages from the network. Available in version 3.16 and later.

To use the default-network-drivers() source, the scl.conf file must be included in your AxoSyslog configuration:

   @include "scl.conf"

Also, make sure that your SELinux, AppArmor, and firewall settings permit AxoSyslog to access the ports where you want to receive messages, and that no other application is using these ports. By default, the default-network-drivers() source accepts messages on the following ports:

  • 514, both TCP and UDP, for RFC3164 (BSD-syslog) formatted traffic
  • 601 TCP, for RFC5424 (IETF-syslog) formatted traffic
  • 6514 TCP, for TLS-encrypted traffic

In addition to receiving messages on different ports and in different formats, this source tries to parse the messages automatically. If successful, it sets the ${.app.name} name-value pair to the name of the application that sent the log message. Currently it uses the following procedures.

Parsing RFC3164-formatted messages

For RFC3164-formatted messages (that is, messages received on the ports set in options udp-port() and tcp-port() which default to port 514), AxoSyslog attempts to use the following parsers. If a parser cannot parse the message, it passes the original message to the next parser.

  1. Parse the incoming raw message as a message from a Cisco device.

  2. Parse the incoming message as an RFC3164-formatted message.

    • If the incoming message was sent by a AxoSyslog client using the syslog-ng() destination, parse its fields as a AxoSyslog message.

      The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving AxoSyslog component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first AxoSyslog component in your infrastructure has received it, which is important if you want to forward a message for example, to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.

    • Otherwise, apply the application adapters if the message was sent from an application that already has a specific parser in AxoSyslog (for example, Splunk Common Information Model (CIM), iptables, or sudo).

Parsing RFC5424-formatted messages

For RFC5424-formatted messages (that is, messages received on the ports set in options rfc5424-tls-port() and rfc5424-tcp-port(), which default to port 601 and 6514), AxoSyslog parses the message according to RFC5424, then attempts apply the application adapters if the message was sent from an application that already has a specific parser in AxoSyslog (for example, Splunk Common Information Model (CIM), iptables, or sudo).

Example: Using the default-network-drivers() driver

The following example uses only the default settings.

   source s_network {
        default-network-drivers();
    };

The following example can receive TLS-encrypted connections on the default port (port 6514).

   source s_network {
        default-network-drivers(
            tls(
                key-file("/path/to/ssl-private-key")
                cert-file("/path/to/ssl-cert")
            )
        );
    };

7.4.1 - default-network-drivers() source options

The systemd-journal() driver has the following options.

ca-dir()

Accepted values:Directory name
Default:none

Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.

This option can be used together with the optional ca-file() option.

ca-file()

Accepted values:File name
Default:empty

Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.

Example format in configuration:

   ca-file("/etc/pki/tls/certs/ca-bundle.crt")

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

log-msg-size()

Type:number (bytes)
Default:Use the global log-msg-size() option, which defaults to 65536 (64 KiB).

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

max-connections()

Type:number
Default:10

Description: Specifies the maximum number of simultaneous connections.

Note that the total number of connections the default-network-drivers() source can use is 3*max-connections(), because this value applies to the network(tcp), syslog(tcp), and syslog(tls) connections individually.

rfc5424-tcp-port()

Type:number
Default:601

Description: The TCP port number where the default-network-drivers() source receives RFC5424-formatted (IETF-syslog) messages.

rfc5424-tls-port()

Type:number
Default:6514

Description: The TCP port number where the default-network-drivers() source receives RFC5424-formatted (IETF-syslog), TLS-encrypted messages.

tcp-port()

Type:number
Default:514

Description: The TCP port number where the default-network-drivers() source receives RFC3164-formatted (BSD-syslog) messages.

tls()

Type:tls options
Default:n/a

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

udp-port()

Type:number
Default:514

Description: The UDP port number where the default-network-drivers() source receives RFC3164-formatted (BSD-syslog) messages.

7.5 - internal: Collect internal messages

All messages generated internally by AxoSyslog use this special source. To collect warnings, errors and notices from AxoSyslog itself, include this source in one of your source statements.

   internal()

The AxoSyslog application will issue a warning upon startup if none of the defined log paths reference this driver.

Example: Using the internal() driver

   source s_local { internal(); };

The AxoSyslog application sends the following message types from the internal() source:

  • fatal: Priority value: critical (2), Facility value: syslog (5)

  • error: Priority value: error (3), Facility value: syslog (5)

  • warning: Priority value: warning (4), Facility value: syslog (5)

  • notice: Priority value: notice (5), Facility value: syslog (5)

  • info: Priority value: info (6), Facility value: syslog (5)

7.5.1 - internal() source options

The internal() driver has the following options:

host-override()

Type:string
Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

log-iw-size()

Type:number
Default:100

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

normalize-hostnames()

Accepted values:yes, no
Default:no

Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.

program-override()

Type:string
Default:

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

use-fqdn()

Type:yes or no
Default:no

Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

7.6 - file: Collect messages from text files

Collects log messages from plain-text files, for example, from the logfiles of an Apache webserver. If you want to use wildcards in the filename, use the wildcard-file() source.

The AxoSyslog application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When AxoSyslog is restarted, it records the position of the last sent log message in the /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist file, and continues to send messages from this position after the restart.

The file driver has a single required parameter specifying the file to open. If you want to use wildcards in the filename, use the wildcard-file() source. For the list of available optional parameters, see file() source options.

Declaration:

   file("filename");

Example: Using the file() driver

   source s_file {
        file("/var/log/messages");
    };

Example: Tailing files

The following source checks the access.log file every second for new messages.

   source s_tail {
        file("/var/log/apache/access.log" follow-freq(1) flags(no-parse));
    };

7.6.1 - Notes on reading kernel messages

Note the following points when reading kernel messages on various platforms.

  • The kernel usually sends log messages to a special file (/dev/kmsg on BSDs, /proc/kmsg on Linux). The file() driver reads log messages from such files. The AxoSyslog application can periodically check the file for new log messages if the follow-freq() option is set.

  • On Linux, the klogd daemon can be used in addition to AxoSyslog to read kernel messages and forward them to syslog-ng. klogd used to preprocess kernel messages to resolve symbols and so on, but as this is deprecated by ksymoops there is really no point in running both klogd and AxoSyslog in parallel. Also note that running two processes reading /proc/kmsg at the same time might result in dead-locks.

  • When using AxoSyslog to read messages from the /proc/kmsg file, AxoSyslog automatically disables the follow-freq() parameter to avoid blocking the file.

  • To read the kernel messages on HP-UX platforms, use the following options in the source statement:

        file("/dev/klog" program-override("kernel") flags(kernel) follow-freq(0));
    

7.6.2 - file() source options

The file() driver has the following options:

default-facility()

Type:facility string
Default:kern

Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.

default-priority()

Type:priority string
Default:

Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).

encoding()

Type:string
Default:

Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

follow-freq()

Type:number
Default:1

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

keep-timestamp()

Type:yes or no
Default:yes

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Type:number
Default:100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Type:number
Default:10000

Description: The size of the initial window, this value is used during flow control. Make sure that log-iw-size() is larger than the value of log-fetch-limit().

log-msg-size()

Type:number (bytes)
Default:Use the global log-msg-size() option, which defaults to 65536 (64 KiB).

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

Type:string
Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

multi-line-garbage()

Type:regular expression
Default:empty string

Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.

To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.

multi-line-mode()

Type:indented, prefix-garbage, prefix-suffix, regexp, smart
Default:empty string

Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:

  • indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.

        source s_tomcat {
            file("/var/log/tomcat/xxx.log" multi-line-mode(indented));
        };
    
  • prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.

  • prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.

    The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.

  • smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.

    smart mode is available in AxoSyslog version 4.2 and newer.

    The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.

multi-line-prefix()

Type:regular expression starting with the ^ character
Default:empty string

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.

Example: Processing Tomcat logs

The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

   2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    SEVERE: Catalina.start:
    LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use null:8080
           at org.apache.catalina.connector.Connector.start(Connector.java:1138)
           at org.apache.catalina.core.StandardService.start(StandardService.java:531)
           at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
           at org.apache.catalina.startup.Catalina.start(Catalina.java:583)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)
    2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    INFO: Server startup in 1206 ms
    2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-8080
    2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
    INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:

source s_file{file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0) multi-line-mode(regexp) multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));};
    };

Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.

multi-line-suffix()

Type:regular expression
Default:empty string

Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.

To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.

pad-size()

Type:number
Default:0

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

program-override()

Type:string
Default:

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Type:name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

7.7 - wildcard-file: Collect messages from multiple text files

The wildcard-file() source collects log messages from multiple plain-text files from multiple directories. The wildcard-file() source is available in AxoSyslog version 3.10 and later.

The AxoSyslog application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When AxoSyslog is restarted, it records the position of the last sent log message in the persist file, and continues to send messages from this position after the restart. The location of the persist file depends on the package you installed AxoSyslog from, typically it is /var/lib/syslog-ng/syslog-ng.persist or /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist.

Declaration:

   wildcard-file(
        base-dir("<pathname>")
        filename-pattern("<filename>")
    );

Note the following important points:

  • You can use the * and ? wildcard characters in the filename (the filename-pattern() option), but not in the path (the base-dir() option).
  • If you use multiple wildcard-file() sources in your configuration, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.

  • When using wildcards, AxoSyslog monitors every matching file (up to the limit set in the max-files() option), and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because AxoSyslog uses the inotify feature of the kernel. Set the max-files() option at least to the number of files you want to monitor. If the wildcard-file source matches more files than the value of the max-files() option, it is random which files will AxoSyslog actually monitor. The default value of max-files() is 100.

  • If the message does not have a proper syslog header, AxoSyslog treats messages received from files as sent by the user facility. Use the default-facility() and default-priority() options in the source definition to assign a different facility if needed.
  • For every message that AxoSyslog reads from the source files, the path and name of the file is available in the ${FILE_NAME} macro.

Required parameters: base-dir(), filename-pattern(). For the list of available optional parameters, see wildcard-file() source options.

Example: Using the wildcard-file() driver

The following example monitors every file with the .log extension in the /var/log directory for log messages.

   source s_files {
        wildcard-file(
            base-dir("/var/log")
            filename-pattern("*.log")
            recursive(no)
            follow-freq(1)
        );
    };

7.7.1 - wildcard-file() source options

The wildcard-file() driver has the following options:

base-dir()

Type:path without filename
Default:

Description: The path to the directory that contains the log files to monitor, for example, base-dir("/var/log"). To monitor also the subdirectories of the base directory, use the recursive(yes) option. For details, see recursive().

   source s_files {
        wildcard-file(
            base-dir("/var/log")
            filename-pattern("*.log")
            recursive(no)
            follow-freq(1)
        );
    };

default-facility()

Type:facility string
Default:kern

Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.

default-priority()

Type:priority string
Default:

Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).

encoding()

Type:string
Default:

Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.

filename-pattern()

Type:filename without path
Default:

Description: The filename to read messages from, without the path. You can use the * and ? wildcard characters, without regular expression and character range support. You cannot use the * and ? literally in the pattern.

For example, filename-pattern("*.log") matches the syslog.log and auth.log files, but does not match the access_log file. The filename-pattern("*log") pattern matches all three.

  • * matches an arbitrary string, including an empty string
  • ? matches an arbitrary character
   source s_files {
        wildcard-file(
            base-dir("/var/log")
            filename-pattern("*.log")
            recursive(no)
            follow-freq(1)
        );
    };

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

follow-freq()

Type:number
Default:1

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

keep-timestamp()

Type:yes or no
Default:yes

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Type:number
Default:100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Type:number
Default:10000

Description: The size of the initial window, this value is used during flow control. Make sure that log-iw-size() is larger than the value of log-fetch-limit().

When using wildcards in the filenames, AxoSyslog attempts to read log-fetch-limit() number of messages from each file. For optimal performance, make sure that log-iw-size() is greater than log-fetch-limit()*max-files(). Note that to avoid performance problems, if log-iw-size()/max-files() is smaller than 100, AxoSyslog automatically sets log-iw-size() to max-files()*100.

Example: Initial window size of file sources

If log-fetch-limit() is 100, and your wildcard file source has 200 files, then log-iw-size() should be at least 20000.

log-msg-size()

Type:number (bytes)
Default:Use the global log-msg-size() option, which defaults to 65536 (64 KiB).

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

Type:string
Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

max-files()

Type:integer
Default:100

Description: Limits the number of files that the wildcard-file source monitors.

When using wildcards, AxoSyslog monitors every matching file (up to the limit set in the max-files() option), and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because AxoSyslog uses the inotify feature of the kernel. Set the max-files() option at least to the number of files you want to monitor. If the wildcard-file source matches more files than the value of the max-files() option, it is random which files will AxoSyslog actually monitor. The default value of max-files() is 100.

monitor-method()

Type:`auto
Default:auto

Description: If the platform supports inotify, AxoSyslog uses it automatically to detect changes to the source files. If inotify is not available, AxoSyslog polls the files as set in the follow-freq() option. To force AxoSyslog poll the files even if inotify is available, set this option to poll.

multi-line-garbage()

Type:regular expression
Default:empty string

Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.

To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.

multi-line-mode()

Type:indented, prefix-garbage, prefix-suffix, regexp, smart
Default:empty string

Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:

  • indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.

        source s_tomcat {
            file("/var/log/tomcat/xxx.log" multi-line-mode(indented));
        };
    
  • prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.

  • prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.

    The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.

  • smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.

    smart mode is available in AxoSyslog version 4.2 and newer.

    The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.

multi-line-prefix()

Type:regular expression starting with the ^ character
Default:empty string

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.

Example: Processing Tomcat logs

The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

   2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    SEVERE: Catalina.start:
    LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use null:8080
           at org.apache.catalina.connector.Connector.start(Connector.java:1138)
           at org.apache.catalina.core.StandardService.start(StandardService.java:531)
           at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
           at org.apache.catalina.startup.Catalina.start(Catalina.java:583)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)
    2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    INFO: Server startup in 1206 ms
    2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-8080
    2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
    INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:

source s_file{file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0) multi-line-mode(regexp) multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));};
    };

Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.

multi-line-suffix()

Type:regular expression
Default:empty string

Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.

To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.

pad-size()

Type:number
Default:0

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

program-override()

Type:string
Default:

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

recursive()

Type:`yes
Default:no

Description: When enabled, AxoSyslog monitors every subdirectory of the path set in the base-dir() option, and reads log messages from files with matching filenames. The recursive option can be used together with wildcards in the filename.

Example: Monitoring multiple directories

The following example reads files having the .log extension from the /var/log/ directory and its subdirectories, including for example, the /var/log/apt/history.log file.

   source s_file_subdirectories {
        wildcard-file(
            base-dir("/var/log")
            filename-pattern("*.log")
            recursive(yes)
            follow-freq(1)
            log-fetch-limit(100)
        );
    };

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Type:name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

7.8 - Hypr Audit Trail and Hypr App Audit Trail

Starting with version 4.2.0, AxoSyslog can fetch events from the Hypr REST API using the hypr-audit-trail() and hypr-app-audit-trail() source drivers.

  • hypr-audit-trail(): is a source driver that pulls messages from the Hypr API, associated to any RP Application ID.
  • hypr-app-audit-trail(): is a source driver that pulls messages from the Hypr API, but only those associated to a specific RP Application ID.

Hypr Audit Trail

The hypr-audit-trail() source queries the Hypr API for the list of potential applications at startup, then monitors the audit trail for each of the detected applications.

Note: Applications that are registered after syslog-ng is started are not recognized.

To start following those audit trails, you must restart syslog-ng.

Example minimal configuration:

source s_hypr {
    hypr-audit-trail(
        url('https://<custom domain>.hypr.com')
        bearer-token('<base64 encoded bearer token>')
    );
};

A more detailed example:

source s_hypr {
    hypr-audit-trail(
        url('https://<custom domain>.hypr.com')
        bearer-token('<base64 encoded bearer token>')
        page-size(<number of results to return in a single page>)
        initial-hours(<number of hours to search backward on initial fetch>)
        application-skip-list('HYPRDefaultApplication', 'HYPRDefaultWorkstationApplication')
        log-level('INFO')
        flags(<optional flags passed to the source>)
        ignore-persistence(<yes/no>)
    );
};

Available options:

  • url(): custom URL for Hypr API access ('https://<custom domain>.hypr.com')
  • bearer-token(): base64 encoded authentication token from Hypr
  • page-size(): number of results to return in a single page (optional - defaults to 100)
  • initial-hours(): number of hours to search backward on initial fetch (optional - defaults to 4)
  • application-skip-list(): list of rpAppIds not to retrieve from Hypr (optional - defaults to 'HYPRDefaultApplication', 'HYPRDefaultWorkstationApplication')
  • log-level(): logging level, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" (optional - defaults to "INFO")
  • flags(): flags passed to the source, can be used for example to disable message parsing with flags(no-parse) (optional - defaults to empty)
  • ignore-persistence(): ignores the saved value in the persist file, and starts querying from the current time (optional - defaults to no)

Hypr App Audit Trail

The hypr-app-audit-trail() monitors the audit trail for one specific RP Application ID. This driver requires the rp-app-id() parameter in order to operate.

Available options:

  • url(): custom URL for Hypr API access ('https://<custom domain>.hypr.com')
  • bearer-token(): base64 encoded authentication token from Hypr
  • rp-app-id(): the RP Application ID for the application to monitor
  • page-size(): number of results to return in a single page (optional - defaults to 100)
  • initial-hours(): number of hours to search backward on initial fetch (optional - defaults to 4)
  • log-level(): logging level, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" (optional - defaults to "INFO")
  • flags(): flags passed to the source, can be used for example to disable message parsing with flags(no-parse) (optional - defaults to empty)
  • ignore-persistence(): ignores the saved value in the persist file, and starts querying from the current time (optional - defaults to no)

Acknowledgements

This documentation page is based on the README file of the hypr-audit-trail() source, written by Dan Elder.

7.9 - Jellyfin logs

Starting with version 4.7.0, you can use the jellyfin() source to read Jellyfin logs from its log file output.

Example minimal configuration:

source s_jellyfin {
    jellyfin(
    base-dir("/path/to/my/jellyfin/root/log/dir")
    filename-pattern("log_*.log")
    );
};

For more details about Jellyfin logging, see:

As the jellyfin() source is based on the wildcard-file() source, you can use the wildcard-file() source options.

This driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

7.10 - linux-audit: Collect messages from Linux audit logs

It reads and automatically parses the Linux audit logs. You can override the file name using the filename() parameter and the prefix for the created name-value pairs using the prefix() parameter. Any additional parameters are passed to the file source.

Declaration:

   linux-audit(options);

Example: Using the linux-audit() driver

source s_auditd {
    linux-audit(
        prefix("test.")
        hook-commands(
            startup("auditctl -w /etc/ -p wa")
            shutdown("auditctl -W /etc/ -p wa")
        )
    );
};
            

7.10.1 - linux-audit() source options

The file() driver has the following options:

filename()

Type:path
Default:

Description: The log file of linux-audit. The AxoSyslog application reads the Linux audit logs from this file.

prefix()

Synopsis:prefix()
Default:.auditd.

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

  • To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

  • To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

  • If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. Note that if you use an empty prefix (prefix("")) or one starting with a dot, AxoSyslog might replace the original value of an existing macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

7.11 - kubernetes: Collect and parse the Kubernetes CRI (Container Runtime Interface) format

The kubernetes() source collects container logs managed by the Kubelet. It reads plain-text and JSON-formatted container logs (as described in the Container Runtime Interface (CRI) design proposal), for example, from the /var/log/containers or /var/log/pods files, and enriches them with various metadata retrieved from the Kubernetes API.

The kubernetes() source is available in AxoSyslog version 3.37 and later.

By default, it reads the /var/log/containers folder and extracts:

  • the log content, and
  • Kubernetes metadata, for example, namespace, pod, and container information.

The Kubernetes-related metadata is available in name-value pairs with the .k8s. prefix. The following table shows the retrieved metadata and their source:

syslog-ng name-value pairsource
.k8s.namespace_nameContainer log file name.
.k8s.pod_nameContainer log file name.
.k8s.pod_uuidContainer log file name or python kubernetes.client.CoreV1Api.
.k8s.container_nameContainer log file name or python kubernetes.client.CoreV1Api.
.k8s.container_idContainer log file name.
.k8s.container_imagepython kubernetes.client.CoreV1Api.
.k8s.container_hashpython kubernetes.client.CoreV1Api.
.k8s.docker_idpython kubernetes.client.CoreV1Api.
.k8s.labels.*python kubernetes.client.CoreV1Api.
.k8s.annotations.*python kubernetes.client.CoreV1Api.

Declaration

   kubernetes(
        base-dir("<pathname>")
    );

7.11.1 - kubernetes() source options

The kubernetes() source has the following options:

base-dir()

Type:path without filename
Default:/var/log/containers

Description: The path to the directory that contains the log files, for example, base-dir("/var/log/pods").

cluster-name()

Type:string
Default:k8s

Description: The name of the Kubernetes cluster.

key-delimiter()

Type:character
Default:.

Description: The delimiter character to use when parsing flattened keys. Supports Only single characters.

prefix()

Synopsis:prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

  • To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

  • To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

  • If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

The prefix() option is optional and its default value is ".k8s.".

7.12 - mbox: Convert local email messages to log messages

Using the mbox() driver, AxoSyslog can read email messages from local mbox files, and convert them to multiline log messages.

This driver has only one required option, the filename of the mbox file. To use the mbox() driver, the scl.conf file must be included in your AxoSyslog configuration:

   @include "scl.conf"

The mbox() driver is actually a reusable configuration snippet configured to read log messages using the file() driver. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the configuration snippet on GitHub.

Example: Using the mbox() driver

The following example reads the emails of the root user on the AxoSyslog host.

   @include "scl.conf"
    source root-mbox {
        mbox("/var/spool/mail/root");
    };

7.12.1 - mbox() source options

The mbox() driver has the following option.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

7.13 - mqtt: receiving messages from an MQTT broker

From AxoSyslog version 3.35, you can use the mqtt() source to fetch messages from MQTT brokers.

The mqtt() source builds on the MQTT protocol, and uses its client and broker entities.

Declaration:

   source s_mqtt{
        mqtt(
            address("tcp://<hostname>:<port-number>")
            topic("<topic-name>")
        );
    };

Starting with AxoSyslog version 4.7, mqtt() source automatically sets the ${MQTT_TOPIC} name-value pair for the messages it receives. This is useful when the name of the topic contains MQTT wildcards ($, +, #). For example:

log {
    source { mqtt(topic("#")); };
    destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); };
};

Example: Using the mqtt() source in your configuration

The following example illustrates an mqtt() source configured to fetch messages from the MQTT broker running on localhost:4444 using the test/test topic, and send them to the localhost:4445 address.

    @version: current
    @include "scl.conf"
    source s_mqtt {
        mqtt(
            address("tcp://localhost:4444")
            topic("test/test")
        );
    };
    destination d_network {
        network(
            "localhost"
            port(4445)
        );
    };
    log {
        source(s_mqtt);
        destination(d_network);;
    };

7.13.1 - Prerequisites to using the mqtt() source

Using the current implementation of the mqtt() source has the following prerequisites:

  • Installing the eclipse-paho-mqtt-c library.

  • Having a broker entity in a functional MQTT system.

7.13.2 - Limitations to using the mqtt() source

Using the mqtt() source of AxoSyslog has the following limitations:

  • You can only use the mqtt() source with AxoSyslog version 3.35 or higher.
  • You cannot use the mqtt() source without installing the the eclipse-paho-mqtt-c library.
  • For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.

  • The current implementation of the mqtt() source supports versions 3.1 and 3.1.1 of the MQTT protocol

7.13.3 - Options of the mqtt() source

The mqtt() source has the following options.

Required options: address(), fallback-topic(), and topic().

address()

Type:string
Default:tcp://localhost:1883
Required:yes

Description: Specifies the hostname or IP address, and the port number of the MQTT broker to which AxoSyslog will send the log messages.

Syntax: <protocol type>://<host>:<port>

client-id()

Type:string
Default:syslog-ng-source-{topic option}
Required:no

Description: The client-id() is used to identify the client to the MQTT server, which stores session data for each client. The session data can contains information regarding which message has been sent, received. It is not possible to set the client-id() to an empty string. To always start a new session see the cleansession() option.

cleansession()

Type:`yes
Default:no

Description: This option instruments the MQTT broker to clean the session data when connecting. The session data contains information about which message was processed.

http-proxy()

Type:URL
Default:N/A

Description: Specifies HTTP/HTTPS proxy for WebSocket connections.

keep-alive()

Type:positive integer number (in seconds)
Default:60

Description: Specifies the number of seconds that AxoSyslog keeps the connection between the broker and clients open in case there is no message traffic. When keep-alive() number of seconds pass, the connection is terminated, and you have to reconnect.

On the MQTT side, the keep alive function provides a workaround method to access connections that are still open.

password()

Type:string
Default:N/A

Description: The password used to authenticate on the MQTT broker.

persist-name()

Type:string
Default:N/A

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

   Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

qos()

Type:number
Default:`0`

Possible values:

`0` - at most once (the fastest option)

`1` - at least once (a much slower option than `0`)

`2` - exactly once (the slowest option)

Description: The Quality of Service (QoS) level in MQTT messaging is an agreement between sender and receiver on the guarantee of delivering a message.

tls()

Type:tls options
Default:n/a

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

The following options are relevant for the mqtt() tls() block: ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), ssl-version(), use-system-cert-store().

topic()

Type:string or template
Default:N/A

Description: Required option. Specifies the MQTT topic.

username()

Type:string
Default:N/A

Description: The username used to authenticate on the MQTT broker.

7.14 - network: Collect messages using the RFC3164 protocol (network() driver)

The network() source driver can receive syslog messages conforming to RFC3164 from the network using the TCP, TLS, and UDP networking protocols.

  • UDP is a simple datagram oriented protocol, which provides “best effort service” to transfer messages between hosts. It may lose messages, and no attempt is made to retransmit lost messages. The BSD-syslog protocol traditionally uses UDP.

    Use UDP only if you have no other choice.

  • TCP provides connection-oriented service: the client and the server establish a connection, each message is acknowledged, and lost packets are resent. TCP can detect lost connections, and messages are lost, only if the TCP connection breaks. When a TCP connection is broken, messages that the client has sent but were not yet received on the server are lost.

  • The AxoSyslog application supports TLS (Transport Layer Security, also known as SSL) over TCP. For details, see Encrypting log messages with TLS.

Declaration:

   network([options]);

By default, the network() driver binds to 0.0.0.0, meaning that it listens on every available IPV4 interface on the TCP/514 port. To limit accepted connections to only one interface, use the localip() parameter. To listen on IPv6 addresses, use the ip-protocol(6) option.

Example: Using the network() driver

Using only the default settings: listen on every available IPV4 interface on the TCP/514 port.

   source s_network {
        network();
    };

UDP source listening on 192.168.1.1 (the default port for UDP is 514):

   source s_network {
        network(
            ip("192.168.1.1")
            transport("udp")
        );
    };

TCP source listening on the IPv6 localhost, port 2222:

   source s_network6 {
        network(
            ip("::1")
            transport("tcp")
            port(2222)
            ip-protocol(6)
        );
    };

A TCP source listening on a TLS-encrypted channel.

   source s_network {
        network(
            transport("tls")
            port(2222)
            tls(peer-verify("required-trusted")
                key-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.key")
                cert-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt")
            );
        );
    };

A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslog messages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses both the IETF-syslog message format and the protocol. For details, see syslog: Collect messages using the IETF-syslog protocol.

   source s_tcp_syslog {
        network(
            ip("127.0.0.1")
            flags(syslog-protocol)
        );
    };

For details on the options of the network() source, see network() source options.

7.14.1 - network() source options

The network() driver has the following options.

ca-dir()

Accepted values:Directory name
Default:none

Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.

This option can be used together with the optional ca-file() option.

ca-file()

Accepted values:File name
Default:empty

Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.

Example format in configuration:

   ca-file("/etc/pki/tls/certs/ca-bundle.crt")

dynamic-window-size()

Type:number
Default:0

Description: The size of the dynamic control window used during flow-control. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

ebpf()

Available in AxoSyslog version 4.2 and newer.

By default, the kernel chooses the receive socket for a specific UDP randomly based on the source IP/port of the sender. You can customize this algorithm using the Extended Berkeley Packet Filter (eBPF) plugin. The ebpf() option changes the kernel’s SO_REUSEPORT algorithm so that all messages are randomly placed into one of the UDP sockets. The decision which UDP socket buffer a datagram is placed is made for every datagram, and not once for every stream. This means that messages are perfectly load-balanced across your set of UDP sockets. While this resolves the imbalance between the sockets and results in perfect load balancing, you will lose ordering between messages from the same sender, which is the price to pay for increased throughput.

source s_network {
    network(
        transport("udp")
        so-reuseport(1) persist-name("udp1")
        ebpf(reuseport(sockets(4)))
    );
    network(transport("udp") so-reuseport(1) persist-name("udp2"));
    network(transport("udp") so-reuseport(1) persist-name("udp3"));
    network(transport("udp") so-reuseport(1) persist-name("udp4"));
};

For a detailed introduction, see the Scaling syslog to 1M EPS with eBPF blog post.

encoding()

Type:string
Default:

Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

  • threaded: The threaded flag enables multithreading for the source. For details on multithreading, see Multithreading and scaling.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

host-override()

Type:string
Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

interface()

Type:string
Default:None

Description: Bind to the specified interface instead of an IP address. Available in 3.19 and later.

ip() or localip()

Type:string
Default:0.0.0.0

Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.

If you specify a multicast bind address and use the udp transport, AxoSyslog automatically joins the necessary multicast group. TCP does not support multicasting.

ip-protocol()

Type:number
Default:4

Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).

Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).

For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.

   source s_network_tcp { network( transport("tcp") ip("::") ip-protocol(6) port(601) ); };

ip-tos()

Type:number
Default:0

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()

Type:number
Default:0

Description: Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Type:yes or no
Default:yes

Description: Specifies whether connections to sources should be closed when AxoSyslog is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the AxoSyslog connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep-hostname()

Type:yes or no
Default:no

Description: Enable or disable hostname rewriting.

  • If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.

  • If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

keep-timestamp()

Type:yes or no
Default:yes

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

listen-backlog()

Type:integer
Default:256

Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:

   source s_network {
        network(
            ip("192.168.1.1")
            transport("tcp")
            listen-backlog(2048)
            );
    };

log-fetch-limit()

Type:number
Default:100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Type:number
Default:100

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

If the max-connections() option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection. For optimal performance when receiving messages from AxoSyslog clients, make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example: Initial window size of a connection

If log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log-msg-size()

Type:number (bytes)
Default:Use the global log-msg-size() option, which defaults to 65536 (64 KiB).

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

max-connections()

Type:number
Default:10

Description: Specifies the maximum number of simultaneous connections.

pad-size()

Type:number
Default:0

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

port() or localport()

Type:number
Default:

In case of TCP transport: 514

In case of UDP transport: 514

Description: The port number to bind to.

program-override()

Type:string
Default:

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

so-broadcast()

Type:yes or no
Default:no

Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.

so-keepalive()

Type:yes or no
Default:no

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-rcvbuf()

Type:number
Default:0

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so-reuseport()

Type:yes or no
Default:no

Description: Enables SO_REUSEPORT on systems that support it. When enabled, the kernel allows multiple UDP sockets to be bound to the same port, and the kernel load-balances incoming UDP datagrams to the sockets. The sockets are distributed based on the hash of (srcip, dstip, srcport, dstport), so the same listener should be receiving packets from the same endpoint. For example:

   source {
            network(transport("udp") so-reuseport(1) port(2000) persist-name("udp1"));
            network(so-reuseport(1) port(2000) persist-name("udp2"));
            network(so-reuseport(1) port(2000) persist-name("udp3"));
            network(so-reuseport(1) port(2000) persist-name("udp4"));
    };

Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-sndbuf()

Type:number
Default:0

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Type:name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

transport()

Type:udp, tcp, tls, proxied-tcp, proxied-tls, proxied-tls-passthrough, text-with-nuls
Default:tcp

Description: Specifies the protocol used to receive messages from the source.

For detailed information about how AxoSyslog supports the proxied-tcp, the proxied-tls, and the proxied-tls-passthrough parameters, see Proxy Protocol support.

text-with-nuls: Allows embedded NUL characters in the message from a TCP source, that is, AxoSyslog will not delimiter the incoming messages on NUL characters, only on newline characters (contrary to tcp transport, which splits the incoming log on newline characters and NUL characters).

trim-large-messages()

Type:yes
Default:Use the global trim-large-messages() option, which defaults to no.

Description: Determines what AxoSyslog does with incoming log messages that are received using the IETF-syslog protocol using the syslog() driver, and are longer than the value of log-msg-size(). Other drivers ignore the trim-large-messages() option.

  • If set to no, AxoSyslog drops the incoming log message.

  • If set to yes, AxoSyslog trims the incoming log message to the size set in log-msg-size(), and adds the trimmed tag to the message. The rest of the message is dropped. You can use the tag to filter on such messages.

        filter f_trimmed {
            tags("trimmed");
        };
    

    If AxoSyslog trims a log message, it sends a debug-level log message to its internal() source.

    As a result of trimming, a parser could fail to parse the trimmed message. For example, a trimmed JSON or XML message will not be valid JSON or XML.

Available in AxoSyslog version 3.21 and later.

Uses the value of the global option if not specified.

tls()

Type:tls options
Default:n/a

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

use-dns()

Type:yes, no, persist_only
Default:yes

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example, from /etc/hosts). The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

use-fqdn()

Type:yes or no
Default:no

Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.

7.14.2 - Proxy Protocol support

If you connect load balancers to your AxoSyslog application, AxoSyslog identifies every connection that is connected to the load balancers identically by default, regardless of the source IP or the source protocol. Essentially, the load balancer masks the source IP unless you enable Proxy Protocol support for your proxy TLS transport() to inject information about the original connection into the forwarded TCP session.

For further details about the working mechanism behind the Proxy Protocol support on AxoSyslog and the configuration details, see the following sections:

7.14.2.1 - The working mechanism behind the Proxy Protocol

This section describes how AxoSyslog supports the Proxy Protocol.

The working mechanism behind the Proxy Protocol

When using the Proxy Protocol during load balancing, AxoSyslog detects the information behind connections connected to the load balancer, then parses the injected information and adds the following macros to every message the comes through the connection later on:

  • PROXY_SRCIP (the source IP of the proxy)

  • PROXY_SRCPORT (the source port of the proxy)

  • PROXY_DSTIP (the destination IP of the proxy)

  • PROXY_DSTPORT (the destination port of the proxy)

7.14.2.2 - Proxy Protocol: configuration and output examples

This section provides information about enabling Proxy Protocol support in your network() source options, and an example configuration and output to illustrate how the Proxy Protocol method works in AxoSyslog.

For more information about the working mechanism of the Proxy Protocol, see The working mechanism behind the Proxy Protocol.

Enabling Proxy Protocol support for your network() source options

Unless you enable Proxy Protocol support for your network() source, AxoSyslog identifies every connection that is connected to the load balancers identically by default, regardless of the source IP or the source protocol.

To enable Proxy Protocol for your network() source, set the transport() parameter of your network() source to proxied-tcp or proxied-tls-passthrough, depending on your preference and configuration.

proxied-tls can be used in complex MITM (man in the middle) configurations, where the proxy header is sent encrypted within the same TLS session as the proxied messages.

When you enable Proxy Protocol support for your network() source, you can use the following configuration example with your AxoSyslog application.

Configuration

The following code sample illustrates how you can use the Proxy Protocol in your AxoSyslog configuration (using the transport() parameter set to proxied-tls-passthrough).

   @version: 3.35
    
    source s_tcp_pp {
      network (
        port(6666)
        transport("proxied-tls-passthrough")
        tls(
            key-file("/certs/certs/server/server.rsa")
            cert-file("/certs/certs/server/server.crt")
            ca-dir("/certs/certs/CA")
            peer-verify("required-trusted")
        )
      );
    };
    
    destination d_file {
      file("/var/log/proxy-proto.log" template("$(format-json --scope nv-pairs)\n"));
    };
    
    log {
      source(s_tcp_pp);
      destination(d_file);
    };

With this configuration, the Proxy Protocol method will perform injecting the information of the original connection into the forwarded TCP session, based on the working mechanism described in The working mechanism behind the Proxy Protocol.

The following example illustrates how the parsed macros will appear in the output.

Example: Output for the PROXY TCP4 192.168.1.1 10.10.0.1 1111 2222 input header

With the PROXY TCP4 192.168.1.1 10.10.0.1 1111 2222 input header, the output looks like this:

   {"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"1111","PROXIED_SRCIP":"192.168.1.1","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"2222","PROXIED_DSTIP":"10.10.0.1","PROGRAM":"TestMsg","MESSAGE":"","LEGACY_MSGHDR":"TestMsg","HOST_FROM":"localhost","HOST":"localhost"}

Note that the macros that AxoSyslog adds to the message appear in the output.

7.15 - nodejs: Receive JSON messages from nodejs applications

Using the nodejs() driver, AxoSyslog can receive application logs directly from nodejs applications that use the widespread Winston logging API. The AxoSyslog application automatically adds the .nodejs.winston. prefix to the name of the fields the extracted from the message.

To use the nodejs() driver, the scl.conf file must be included in your AxoSyslog configuration:

   @include "scl.conf"

The nodejs() driver is actually a reusable configuration snippet configured to receive log messages using the network() driver, and process its JSON contents. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the nodejs configuration snippet on GitHub.

Example: Using the nodejs() driver

The following example uses the default settings of the driver, listening for messages on port 9003 of every IP address of the AxoSyslog host.

   @include "scl.conf"
    source apps { nodejs(); };

The following example listens only on IP address 192.168.1.1, port 9999.

   @include "scl.conf"
    source apps {
        nodejs(
            localip(192.168.1.1)
            port(9999)
        )
    };

7.15.1 - nodejs() source options

The nodejs() driver has the following options.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

ip() or localip()

Type:string
Default:0.0.0.0

Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.

If you specify a multicast bind address and use the udp transport, AxoSyslog automatically joins the necessary multicast group. TCP does not support multicasting.

port() or localport()

Type:number
Default:9003

Description: The port number to bind to.

7.16 - osquery: Collect and parse osquery result logs

The osquery application allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. You can make queries on demand, and also schedule them to run regularly.

The osquery() source of AxoSyslog allows you read the results of periodical osquery queries (from the /var/log/osquery/osqueryd.results.log file) and automatically parse the messages (if you want to use AxoSyslog to send log messages to osquery, read this blogpost). For example, you can:

  • Create filters from the fields of the messages.

  • Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on).

  • Send the messages to a central location, for example, to Elasticsearch, directly from AxoSyslog.

The AxoSyslog application automatically adds the .osquery. prefix to the name of the fields the extracted from the message.

The osquery() source is available in AxoSyslog version 3.10 and later.

Prerequisites:

  • To use the osquery() driver, the scl.conf file must be included in your AxoSyslog configuration:

        @include "scl.conf"
    
  • AxoSyslog must be compiled with JSON-support enabled.

The osquery() driver is actually a reusable configuration snippet configured to read the osquery log file using the file() driver, and process its JSON contents. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

Example: Using the osquery() driver with the default settings

The following AxoSyslog configuration sample uses the default settings of the driver, reading osquery result logs from the /var/log/osquery/osqueryd.results.log file, and writes the log messages generated from the traps into a file.

   @version: 3.10
    @include "scl.conf"
    source s_osquery {
        osquery();
    };
    log {
        source(s_osquery);
        destination {
            file("/var/log/example.log");
        };
    };

Filter for messages related to loading Linux kernel modules:

   @version: 3.10
    @include "scl.conf"
    source s_osquery {
        osquery();
    };
    log {
        source(s_osquery);
        filter f_modules {
            "${.osquery.name}" eq "pack_incident-response_kernel_modules"
        };
        destination {
            file("/var/log/example.log");
        };
    };

Example: Using the osquery() driver with custom configuration

The following AxoSyslog configuration sample reads osquery result logs from the /tmp/osquery_input.log file, and writes the log messages generated from the traps into a file. Using the format-json template, the outgoing message will be a well-formed JSON message.

Input message:

   {"name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","calendarTime":"Fri Jul 21 10:04:41 2017 UTC","unixTime":"1500631481","decorations":{"host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","username":"myuser"},"columns":{"build_distro":"xenial","build_platform":"ubuntu","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","config_valid":"1","counter":"4","extensions":"active","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","pid":"19764","resident_size":"26416000","start_time":"1500629552","system_time":"223","user_time":"476","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","version":"2.5.0","watcher":"19762"},"action":"added"}

AxoSyslog configuration:

   @version: 3.10
    @include "scl.conf"
    source s_osquery {
        osquery(
          file(/tmp/osquery_input.log)
          prefix(.osquery.)
        );
    };
    destination d_file {
        file(
          "/tmp/output.txt"
          template("$(format_json --key .osquery.*)\n")
        );
    };
    log {
      source(s_osquery);
      destination(d_file);
      flags(flow-control);
    };

Outgoing message:

   Outgoing message; message='{"_osquery":{"unixTime":"1500631481","name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","user_time":"476","system_time":"223","start_time":"1500629552","resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","extensions":"active","counter":"4","config_valid":"1","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu","build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a'

To configure a destination to send the log messages to Elasticsearch, see elasticsearch2: DEPRECATED - Send messages directly to Elasticsearch version 2.0 or higher. For other destinations, see destination: Forward, send, and store log messages.

7.16.1 - osquery() source options

The osquery() driver has the following options.

file()

Type:path
Default:/var/log/osquery/osqueryd.results.log

Description: The log file of osquery that stores the results of periodic queries. The AxoSyslog application reads the messages from this file.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

prefix()

Synopsis:prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

  • To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

  • To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

  • If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

Default value:

.osquery. option.

7.17 - Receive logs, metrics, and traces from OpenTelemetry

Starting with version 4.3.0, AxoSyslog can receive logs, metrics, and traces from OpenTelemetry clients over the OpenTelemetry Protocol (OTLP/gRPC).

Example: Receiving OpenTelemetry data

The following example receives OpenTelemetry data and forwards it to an OpenTelemetry receiver. Note that by default, AxoSyslog doesn’t parse the fields of the incoming messages into name-value pairs, but are only available for forwarding using the opentelemetry() destination. To parse the fields into name-value pairs, use the opentelemetry() parser.

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(4317)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

auth()

You can set authentication in the auth() option of the driver. By default, authentication is disabled (auth(insecure())).

The following authentication methods are available in the auth() block:

adc()

Application Default Credentials (ADC). This authentication method is only available for destinations.

alts()

Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.

source {
    opentelemetry(
      port(4317)
      auth(alts())
    );
  };
destination {
    loki(
      port(12345)
      auth(alts())
    );
  };
source {
    syslog-ng-otlp(
      port(4317)
      auth(alts())
    );
  };

insecure()

This is the default method, authentication is disabled (auth(insecure())).

tls()

tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.

destination {
    opentelemetry(
      url("your-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
destination {
    loki(
      url("your-loki-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
destination {
    syslog-ng-otlp(
      url("your-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };

Note:

  • tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
  • The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.

channel-args()

Type:See description
Default:-

Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:

    channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
    )

concurrent-requests()

Type:integer
Default:2

Description: Configures the maximal number of in-flight gRPC requests per worker. Setting this value in the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater than or equal to the number of clients, but this can increase the memory usage.

log-fetch-limit()

Type:number
Default:100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

port()

The port number to receive incoming connections. Default value: 4317

workers()

Type:integer
Default:1

Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to process the messages of the source. Increasing the number of worker threads can drastically improve the performance of the source.

7.18 - pacct: Collect process accounting logs on Linux

Starting with version 3.2, AxoSyslog can collect process accounting logs on Linux systems.Process accounting is the method of recording and summarizing commands executed on Linux, for example, the commands being run, the user who executed the command, CPU time used by the process, exit code, and so on. When process accounting (also called pacct) is enabled on a system, the kernel writes accounting records to the /var/log/account/pacct file (might vary between different Linux distributions).

To use the pacct() driver, the following conditions must be met:

  • The AxoSyslog application must be compiled with the --enable-pacct option. Execute the syslog-ng -V command to check if your binary supports process accounting.

  • The pacctformat plugin must be loaded. By default, AxoSyslog automatically loads the available modules.

  • The scl.conf file must be included in your AxoSyslog configuration:

        @include "scl.conf"
    
  • Process accounting must be running on the host. You can enable it with the accton command.

The pacct() driver parses the fields of the accounting logs and transforms them into name-value pairs. The fields are defined in the manual page of the accounting log file (man acct), AxoSyslog prepends every field with the .pacct. prefix. For example, the ac_uid field that contains the id of the user who started the process will be available under the $.pacct.ac_uid name. These can be used as macros in templates, in filters to select specific messages, and so on.

To use the pacct() driver, use the following syntax.

   @version: 4.5.0
    @include "scl.conf"
    source s_pacct { pacct(); };
    ...
    log { source(s_pacct); destination(...); };

The pacct() driver is actually a reusable configuration snippet configured to handle Linux accounting logs. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the pacct configuration snippet on GitHub.

7.18.1 - pacct() options

The pacct() driver has the following options:

file()

Type:filename with path
Default:/var/log/account/pacct

Description: The file where the process accounting logs are stored — AxoSyslog reads accounting messages from this file.

follow-freq()

Type:number
Default:1

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

7.19 - Pi-hole Faster Than Light logs

Starting with version 4.6.0, AxoSyslog can collect logs of the Pi-hole FTL (Faster Than Light) application.

source s_pihole_ftl {
  pihole-ftl();
};

By default, the source reads the /var/log/pihole/FTL.log file. If the root directory of your Pi-hole installation is different, specify the directory where the FTL.log file is with the dir() option. You can find the root log directory by selecting Tools > Pi-hole diagnosis in the Pi-hole application. Otherwise, the pihole-ftl() source has the same parameters as the file() source.

The pihole-ftl() driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

7.20 - pipe: Collect messages from named pipes

The pipe driver opens a named pipe with the specified name and listens for messages. It is used as the native message delivery protocol on HP-UX.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. For the list of available optional parameters, see pipe() source options.

Declaration:

   pipe(filename);

Pipe is very similar to the file() driver, but there are a few differences, for example, pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg.

Example: Using the pipe() driver

   source s_pipe {
        pipe("/dev/pipe" pad-size(2048));
    };

7.20.1 - pipe() source options

The pipe driver has the following options:

create-dirs()

Type:yes or no
Default:no

Description: Enable creating non-existing directories when creating files or socket files.

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

follow-freq()

Type:number
Default:1

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

keep-timestamp()

Type:yes or no
Default:yes

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Type:number
Default:100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Type:number
Default:100

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

log-msg-size()

Type:number (bytes)
Default:Use the global log-msg-size() option, which defaults to 65536 (64 KiB).

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

Type:string
Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

multi-line-garbage()

Type:regular expression
Default:empty string

Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.

To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.

multi-line-mode()

Type:indented, prefix-garbage, prefix-suffix, regexp, smart
Default:empty string

Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:

  • indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.

        source s_tomcat {
            file("/var/log/tomcat/xxx.log" multi-line-mode(indented));
        };
    
  • prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.

  • prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.

    The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.

  • smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.

    smart mode is available in AxoSyslog version 4.2 and newer.

    The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.

multi-line-prefix()

Type:regular expression starting with the ^ character
Default:empty string

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.

Example: Processing Tomcat logs

The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

   2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    SEVERE: Catalina.start:
    LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use null:8080
           at org.apache.catalina.connector.Connector.start(Connector.java:1138)
           at org.apache.catalina.core.StandardService.start(StandardService.java:531)
           at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
           at org.apache.catalina.startup.Catalina.start(Catalina.java:583)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)
    2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    INFO: Server startup in 1206 ms
    2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-8080
    2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
    INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:

source s_file{file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0) multi-line-mode(regexp) multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));};
    };

Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.

multi-line-suffix()

Type:regular expression
Default:empty string

Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.

To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.

optional()

Type:yes or no
Default:

Description: Instruct AxoSyslog to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.

pad-size()

Type:number
Default:0

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

program-override()

Type:string
Default:

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Type:name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

7.21 - program: Receive messages from external applications

The program driver starts an external application and reads messages from the standard output (stdout) of the application. It is mainly useful to receive log messages from daemons that accept incoming messages and convert them to log messages.

The program driver has a single required parameter, specifying the name of the application to start.

Declaration:

   program(filename);

Example: Using the program() driver

   source s_program {
        program("/etc/init.d/mydaemon");
    };

7.21.1 - program() source options

The program driver has the following options:

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

keep-timestamp()

Type:yes or no
Default:yes

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Type:number
Default:100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

inherit-environment()

Type:yes
Default:yes

Description: By default, when program() starts an external application or script, it inherits the entire environment of the parent process (that is, AxoSyslog). Use inherit-environment(no) to prevent this.

log-iw-size()

Type:number
Default:100

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

log-msg-size()

Type:number (bytes)
Default:Use the global log-msg-size() option, which defaults to 65536 (64 KiB).

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

Type:string
Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

optional()

Type:yes or no
Default:

Description: Instruct AxoSyslog to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.

pad-size()

Type:number
Default:0

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

program()

Type:filename with path
Default:

Description: The name of the application to start and read messages from.

program-override()

Type:string
Default:

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Type:name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

7.22 - python: writing server-style Python sources

The Python source allows you to write your own source in Python. You can import external Python modules to receive or fetch the messages. Since many services have a Python library, the Python source makes integrating AxoSyslog very easy and quick.

You can write two different type of sources in Python:

  • Server-style sources that receives messages. Write server-style sources if you want to use an event-loop based, nonblocking server framework in Python, or if you want to implement a custom loop.

  • Fetcher-style sources that actively fetch messages. In general, write fetcher-style sources (for example, when using simple blocking APIs), unless you explicitly need a server-style source.

This section describes server-style sources. For details on fetcher-style sources, see python-fetcher: writing fetcher-style Python sources.

The following points apply to using Python blocks in AxoSyslog in general:

  • Python parsers and template functions are available in AxoSyslog version 3.10 and later.

    Python destinations and sources are available in AxoSyslog version 3.18 and later.

  • Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).

  • The Python block must be a top-level block in the AxoSyslog configuration file.

  • If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.

    In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".

  • The Python object is initiated every time when AxoSyslog is started or reloaded.

  • The Python block can contain multiple Python functions.

  • Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.

  • Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.

  • Python error messages are available in the internal() source of AxoSyslog.

  • You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.

  • To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.

Declaration:

Python sources consist of two parts. The first is a AxoSyslog source object that you define in your AxoSyslog configuration and use in the log path. This object references a Python class, which is the second part of the Python source. The Python class receives or fetches the log messages, and can do virtually anything that you can code in Python. You can either embed the Python class into your AxoSyslog configuration file, or store it in an external Python file.

   source <name_of_the_python_source>{
        python(
            class("<name_of_the_python_class_executed_by_the_source>")
            options(
                "option1" "value1",
                "option2" "value2"
            )
        );
    };
    
    python {
    from syslogng import LogSource
    from syslogng import LogMessage
    
    class <name_of_the_python_class_executed_by_the_source>(LogSource):
        def init(self, options): # optional
            print("init")
            print(options)
            self.exit = False
            return True
    
        def deinit(self): # optional
            print("deinit")
    
        def run(self): # mandatory
            print("run")
            while not self.exit:
                # Must create a message
                msg = LogMessage("this is a log message")
                self.post_message(msg)
    
        def request_exit(self): # mandatory
            print("exit")
            self.exit = True
    };

Methods of the python() source

Server-style Python sources must be inherited from the syslogng.LogSource class, and must implement at least the run and request_exit methods. Multiple inheritance is allowed, but only for pure Python super classes.

You can implement your own event loop, or integrate the event loop of an external framework or library, for example, KafkaConsumer, Flask, Twisted engine, and so on.

To post messages, call LogSource::post_message() method in the run method.

For the list of available optional parameters, see python() and python-fetcher() source options.

init(self, options) method (optional)

The AxoSyslog application initializes Python objects every time when it is started or reloaded. The init method is executed as part of the initialization. You can perform any initialization steps that are necessary for your source to work.

When this method returns with False, AxoSyslog does not start. It can be used to check options and return False when they prevent the successful start of the source.

options: This optional argument contains the contents of the options() parameter of the AxoSyslog configuration object as a Python dictionary.

run(self) method (mandatory)

Use the run method to implement an event loop, or start a server framework or library. Create LogMessage instances in this method, and pass them to the log paths by calling LogSource::post_message().

Currently, run stops permanently if an unhandled exception happens.

For details on parsing and posting messages, see Python LogMessage API.

request_exit(self) method (mandatory)

The AxoSyslog application calls this method when AxoSyslog is shut down or restarted. The request_exit method must shut down the event loop or framework, so the run method can return gracefully. If you use blocking operations within the run() method, use request_exit() to interrupt those operations and set an exit flag, otherwise AxoSyslog is not able to stop. Note that AxoSyslog calls the request_exit method from a thread different from the source thread.

close_batch(self)

Closes the current source-side batch. Source-side batching helps AxoSyslog to effectively process a larger chunk of messages, instead of processing messages each message. For example, when feeding a destination queue and instead of taking a lock on the queue for every message (causing contention), we only take it once per batch.

The native drivers built into AxoSyslog typically close batches once every mainloop iteration, allowing a single iteration to process multiple messages. For instance, when receiving multiple messages in a single TCP datagram, all of those messages can be processed as a part of the same batch.

In Python-based log sources, a batch will automatically be closed after every message posted via post_message(), except if self.auto_close_batches is set to False during initialization. In case self.auto_close_batches is set to False, the driver has to call close_batch() explicitly, preferably at a natural boundary between incoming batches of messages. A good example is when we retrieve several messages via the same HTTP REST call, then the right time to close the batch would be after the last message in the response is posted.

The deinit(self) method (optional)

This method is executed when AxoSyslog is stopped or reloaded. This method does not return a value.

set_transport_name(self, name)

Set the transport name used to retrieve messages This function can be called to customize the ${TRANSPORT} name-value pair.

7.22.1 - Python LogMessage API

The LogMessage API allows you to create LogMessage objects in Python sources, parse syslog messages, and set the various fields of the log message.

LogMessage() method: Create log message objects

You can use the LogMessage() method to create a structured log message instance. For example:

   from syslogng import LogMessage
    
    msg = LogMessage() # Initialize an empty message with default values (recvd timestamp, rcptid, hostid, ...)
    msg = LogMessage("string or bytes-like object") # Initialize a message and set its ${MESSAGE} field to the specified argument

You can also explicitly set the different values of the log message. For example:

   msg["MESSAGE"] = "message"
    msg["HOST"] = "hostname"

You can set certain special field (timestamp, priority) by using specific methods.

Note the following points when creating a log message:

  • When setting the hostname, AxoSyslog takes the following hostname-related options of the configuration into account: chain-hostnames(), keep-hostname(), use-dns(), and use-fqdn().

  • Python sources ignore the log-msg-size() option.

  • The AxoSyslog application accepts only one message from every LogSource::post_message() or fetch() call, batching is currently not supported. If your Python code accepts batches of messages, you must pass them to AxoSyslog one-by-one. Similarly, if you need to split messages in the source, you must do so in your Python code, and pass the messages separately.

  • Do not reuse or store LogMessage objects after posting (calling post_message()) or returning the message from fetch().

parse() method: Parse syslog messages

The parse() method allows you to parse incoming messages as syslog messages. By default, the parse() method attempts to parse the message as an IETF-syslog (RFC5424) log message. If that fails, it parses the log message as a BSD-syslog (RFC3164) log message. Note that AxoSyslog takes the parsing-related options of the configuration into account: flags(), keep-hostname(), recv-time-zone().

If keep-hostname() is set to no, AxoSyslog ignores the hostname set in the message, and uses the IP address of the AxoSyslog host as the hostname (to use the hostname instead of the IP address, set the use-dns() or use-fqdn() options in the Python source).

   msg_ietf = LogMessage.parse('<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry', self.parse_options)
    msg_bsd = LogMessage.parse('<34>Oct 11 22:14:15 mymachine su: \'su root\' failed for lonvick on /dev/pts/8', self.parse_options)

set_pri() method

You can set the priority of the message with the set_pri() method.

   msg.set_pri(165)

set_timestamp() method

You can use the set_timestamp() method to set the date and time of the log message.

   timestamp = datetime.fromisoformat("2018-09-11T14:49:02.100+02:00")
    msg.set_timestamp(timestamp) # datetime object, includes timezone information

In Python 2, timezone information cannot be attached to the datetime instance without using an external library. The AxoSyslog represents naive datetime objects in UTC.

In Python 3, naive and timezone-aware datetime objects are both supported.

7.22.2 - python() and python-fetcher() source options

The python() and python-fetcher() drivers have the following options.

class()

Type:string
Default:N/A

Description: The name of the Python class that implements the source, for example:

   python(
        class("MyPythonSource")
    );

If you want to store the Python code in an external Python file, the class() option must include the name of the Python file containing the class, without the path and the .py extension, for example:

   python(
        class("MyPythonfilename.MyPythonSource")
    );

For details, see Python code in external files

fetch-no-data-delay()

Type:integer [seconds]
Default:-1 (disabled)

Description: If the fetch method of a python-fetcher() source returns with the LogFetcher.FETCH_NO_DATA constant, the source waits fetch-no-data-delay() seconds before calling the fetch method again. If you want to call the fetch method sooner, set the fetch-no-data-delay() option to the number of seconds to wait before calling the fetch method.

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

The flags and the hostname-related options (for example, use-dns) set in the configuration file influence the behavior of the LogMessage.parse() method of the Python source. They have no effect if you set the message or the hostname directly, without using LogMessage.parse().

keep-hostname()

Type:yes or no
Default:no

Description: Enable or disable hostname rewriting.

  • If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.

  • If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-iw-size()

Type:number
Default:100

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

loaders()

Type:list of python modules
Default:empty list

Description: The AxoSyslog application imports Python modules specified in this option, before importing the code of the Python class. This option has effect only when the Python class is provided in an external Python file. This option has no effect when the Python class is provided within the AxoSyslog configuration file (in a python{} block). You can use the loaders() option to modify the import mechanism that imports Python class. For example, that way you can use hy in your Python class.

   python(class(usermodule.HyParser) loaders(hy))

options()

Type:string
Default:N/A

Description: This option allows you to pass custom values from the configuration file to the Python code. Enclose both the option names and their values in double-quotes. The Python code will receive these values during initialization as the options dictionary. For example, you can use this to set the IP address of the server from the configuration file, so it is not hard-coded in the Python object.

   python(
        class("MyPythonClass")
        options(
            "host" "127.0.0.1"
            "port" "1883"
            "otheroption" "value")
    );

For example, you can refer to the value of the host field in the Python code as options["host"]. Note that the Python code receives the values as strings, so you might have to cast them to the type required, for example: int(options["port"])

persist-name()

Type:string
Default:N/A

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

   Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-reopen()

Accepted values:number [seconds]
Default:1

Description: The time to wait in seconds before a dead connection is reestablished.

time-zone()

Type:name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

7.23 - python-fetcher: writing fetcher-style Python sources

The Python source allows you to write your own source in Python. You can import external Python modules to receive or fetch the messages. Since many services have a Python library, the Python source makes integrating AxoSyslog very easy and quick.

You can write two different type of sources in Python:

  • Server-style sources that receives messages. Write server-style sources if you want to use an event-loop based, nonblocking server framework in Python, or if you want to implement a custom loop.

  • Fetcher-style sources that actively fetch messages. In general, write fetcher-style sources (for example, when using simple blocking APIs), unless you explicitly need a server-style source.

This section describes fetcher-style sources. For details on server-style sources, see python: writing server-style Python sources.

The following points apply to using Python blocks in AxoSyslog in general:

  • Python parsers and template functions are available in AxoSyslog version 3.10 and later.

    Python destinations and sources are available in AxoSyslog version 3.18 and later.

  • Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).

  • The Python block must be a top-level block in the AxoSyslog configuration file.

  • If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.

    In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".

  • The Python object is initiated every time when AxoSyslog is started or reloaded.

  • The Python block can contain multiple Python functions.

  • Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.

  • Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.

  • Python error messages are available in the internal() source of AxoSyslog.

  • You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.

  • To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.

Declaration:

Python sources consist of two parts. The first is a AxoSyslog source object that you define in your AxoSyslog configuration and use in the log path. This object references a Python class, which is the second part of the Python source. The Python class receives or fetches the log messages, and can do virtually anything that you can code in Python. You can either embed the Python class into your AxoSyslog configuration file, or store it in an external Python file.

   source <name_of_the_python_source>{
        python-fetcher(
            class("<name_of_the_python_class_executed_by_the_source>")
        );
    };
    
    python {
    from syslogng import LogFetcher
    from syslogng import LogMessage
    
    class <name_of_the_python_class_executed_by_the_source>(LogFetcher):
        def init(self, options): # optional
            print("init")
            print(options)
            return True
    
        def deinit(self): # optional
            print("deinit")
    
        def open(self): # optional
            print("open")
            return True
    
        def fetch(self): # mandatory
            print("fetch")
            # return LogFetcher.FETCH_ERROR,
            # return LogFetcher.FETCH_NOT_CONNECTED,
            # return LogFetcher.FETCH_TRY_AGAIN,
            # return LogFetcher.FETCH_NO_DATA,
            return LogFetcher.FETCH_SUCCESS, msg
    
        def request_exit(self):
            print("request_exit")
            # If your fetching method is blocking, do something to break it
            # For example, if it reads a socket: socket.shutdown()
    
        def close(self): # optional
            print("close")
    };

Methods of the python-fetcher() source

Fetcher-style Python sources must be inherited from the syslogng.LogFetcher class, and must implement at least the fetch method. Multiple inheritance is allowed, but only for pure Python super classes.

For fetcher-style Python sources, AxoSyslog handles the event loop and the scheduling automatically. You can use simple blocking server/client libraries to receive or fetch logs.

You can retrieve messages using the fetch() method.

init(self, options) method (optional)

The AxoSyslog application initializes Python objects every time when it is started or reloaded. The init method is executed as part of the initialization. You can perform any initialization steps that are necessary for your source to work.

When this method returns with False, AxoSyslog does not start. It can be used to check options and return False when they prevent the successful start of the source.

options: This optional argument contains the contents of the options() parameter of the AxoSyslog configuration object as a Python dictionary.

open(self) method (optional)

The open(self) method opens the resources required for the source, for example, it initiates a connection to the target service. It is called after init() when AxoSyslog is started or reloaded. If fetch() returns with an error, AxoSyslog calls the close() and open() methods before trying to fetch a new message.

If open() fails, it should return the False value. In this case, AxoSyslog retries it every time-reopen() seconds. By default, this is 1 second for Python sources and destinations, the value of time-reopen() is not inherited from the global option. For details, see Error handling in the python() destination.

fetch(self) method (mandatory)

Use the fetch method to fetch messages and pass them to the log paths.

For details on parsing messages, see Python LogMessage API.

The fetch method must return one of the following values:

  • LogFetcher.FETCH_ERROR: Fetching new messages failed, AxoSyslog calls the close and open methods.

  • LogFetcher.FETCH_NO_DATA: There was not any data available. The source waits before calling the fetch method again. The wait time is equal to time-reopen() by default, but you can override it by setting the fetch-no-data-delay() option in the source.

  • LogFetcher.FETCH_NOT_CONNECTED: Could not access the source, AxoSyslog calls the open method.

  • LogFetcher.FETCH_SUCCESS, msg: Post the message returned as the second argument.

  • LogFetcher.FETCH_TRY_AGAIN: The fetcher could not provide a message this time, but will make the source call the fetch method as soon as possible.

request_exit(self) method (optional)

If you use blocking operations within the fetch() method, use request_exit() to interrupt those operations (for example, to shut down a socket), otherwise AxoSyslog is not able to stop. Note that AxoSyslog calls the request_exit method from a thread different from the source thread.

close(self) method (optional)

Close the connection to the target service. Usually it is called right before deinit() when stopping or reloading AxoSyslog. It is also called when fecth() fails.

close_batch(self)

Closes the current source-side batch. Source-side batching helps AxoSyslog to effectively process a larger chunk of messages, instead of processing messages each message. For example, when feeding a destination queue and instead of taking a lock on the queue for every message (causing contention), we only take it once per batch.

The native drivers built into AxoSyslog typically close batches once every mainloop iteration, allowing a single iteration to process multiple messages. For instance, when receiving multiple messages in a single TCP datagram, all of those messages can be processed as a part of the same batch.

In Python-based log sources, a batch will automatically be closed after every message posted via post_message(), except if self.auto_close_batches is set to False during initialization. In case self.auto_close_batches is set to False, the driver has to call close_batch() explicitly, preferably at a natural boundary between incoming batches of messages. A good example is when we retrieve several messages via the same HTTP REST call, then the right time to close the batch would be after the last message in the response is posted.

The deinit(self) method (optional)

This method is executed when AxoSyslog is stopped or reloaded. This method does not return a value.

For the list of available optional parameters, see python() and python-fetcher() source options.

7.24 - qBittorrent logs

Starting with version 4.6.0, AxoSyslog can collect logs of the qBittorrent application.

source s_qbittorrent {
  qbittorrent(
    dir("/path/to/qbittorrent-root-log-dir")
  );
};

To configure the source, you only have to specify the root log directory of qBittorrent in the dir() parameter. You can find the root log directory by selecting Tools > Preferences > Behavior > Log file > Save path in the qBittorrent application. Otherwise, the qbittorrent() source has the same parameters as the file() source.

The qbittorrent() driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

7.25 - snmptrap: Read Net-SNMP traps

Using the snmptrap() source, you can read and parse the SNMP traps of the Net-SNMP’s snmptrapd application. AxoSyslog can read these traps from a log file, and extract their content into name-value pairs, making it easy to forward them as a structured log message (for example, in JSON format). The AxoSyslog application automatically adds the .snmp. prefix to the name of the fields the extracted from the message.

The snmptrap() source is available in AxoSyslog version 3.10 and later.

Limitations:

  • The snmptrap() source has only the options listed in snmptrap() source options. Other options commonly available in other source drivers are not supported.

  • In addition to traps, the log of snmptrapd may contain other messages (for example, daemon start/stop information, debug logs) as well. Currently AxoSyslog discards these messages.

  • The AxoSyslog application cannot resolve OIDs, you have to configure snmptrapd to do so. Note that because of a bug, if snmptrapd does not escape String values in the VarBindList if it can resolve an OID to a symbolic name. As a result, AxoSyslog cannot process traps that contain the = in the value of the string. To overcome this problem, disable resolving OIDs in snmptrapd. For details, see the documentation of snmptrapd.

  • The colon (:) character is commonly used in SNMP traps. However, this character cannot be used in the name of AxoSyslog macros (name-value pairs). Therefore, the AxoSyslog application automatically replaces all consecutive : characters with a single underscore (_) character. For example, you can reference the value of the NET-SNMP-EXAMPLES-MIB::netSnmpExampleString key using the ${NET-SNMP-EXAMPLES-MIB_netSnmpExampleString} macro.

    Note that this affects only name-value pairs (macros). The generated message always contains the original name of the key.

Prerequisites:

  • Configure snmptrapd to log into a file.

  • If you use SMIv1 traps, include the following format string in the configuration file of snmptrapd:

        format1 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]: %N\n\t%W Trap (%q) Uptime: %#T\n%v\n
    
  • If you use SMIv2 traps, use the default format. The snmptrap() source of AxoSyslog expects this default format:

        format2 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]:\n%v\n
    
  • Beacause of an snmptrapd bug, if you specify the filename in the configuration file with logOption, you must also specify another output as a command line argument (-Lf, -Ls). Otherwise, snmptrapd will not apply the the trap format.

To use the snmptrap() driver, the scl.conf file must be included in your AxoSyslog configuration:

   @include "scl.conf"

Example: Using the snmptrap() driver

A sample snmptrapd configuration:

   authCommunity log,execute,net public
    format1 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]: %N\n\t%W Trap (%q) Uptime: %#T\n%v\n
    outputOption s

Starting snmptrapd: snmptrapd -A -Lf /var/log/snmptrapd.log

Sending a sample V2 trap message: snmptrap -v2c -c public 127.0.0.1 666 NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification netSnmpExampleHeartbeatRate i 60 netSnmpExampleString s "string". From this trap, AxoSyslog receives the following input:

   2017-05-23 15:29:40 localhost [UDP: [127.0.0.1]:59993->[127.0.0.1]:162]:
    SNMPv2-SMI::mib-2.1.3.0 = Timeticks: (666) 0:00:06.66   SNMPv2-SMI::snmpModules.1.1.4.1.0 = OID: NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification     NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate = INTEGER: 60        NET-SNMP-EXAMPLES-MIB::netSnmpExampleString = STRING: string

The following AxoSyslog configuration sample uses the default settings of the driver, reading SNMP traps from the /var/log/snmptrapd.log file, and writes the log messages generated from the traps into a file.

   @include "scl.conf"
    log {
      source {
        snmptrap(filename("/var/log/snmptrapd.log"));
      };
      destination {
        file("/var/log/example.log");
      };
    };

From the trap, AxoSyslog writes the following into the log file:

   May 23 15:29:40 myhostname snmptrapd: hostname='localhost', transport_info='UDP: [127.0.0.1]:59993->[127.0.0.1]:162', SNMPv2-SMI::mib-2.1.3.0='(666) 0:00:06.66', SNMPv2-SMI::snmpModules.1.1.4.1.0='NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification', NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate='60', NET-SNMP-EXAMPLES-MIB::netSnmpExampleString='string'

Using the same input trap, the following configuration example formats the SNMP traps as JSON messages.

   @include "scl.conf"
    log {
      source {
        snmptrap(
          filename("/var/log/snmptrapd.log")
          set-message-macro(no)
        );
      };
    
      destination {
        file("/var/log/example.log" template("$(format-json --scope dot-nv-pairs)\n"));
      };
    };

The previous trap formatted as JSON:

   {
       "_snmp":{
          "transport_info":"UDP: [127.0.0.1]:59993->[127.0.0.1]:162",
          "hostname":"localhost",
          "SNMPv2-SMI_snmpModules":{
             "1":{
                "1":{
                   "4":{
                      "1":{
                         "0":"NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification"
                      }
                   }
                }
             }
          },
          "SNMPv2-SMI_mib-2":{
             "1":{
                "3":{
                   "0":"(666) 0:00:06.66"
                }
             }
          },
          "NET-SNMP-EXAMPLES-MIB_netSnmpExampleString":"string",
          "NET-SNMP-EXAMPLES-MIB_netSnmpExampleHeartbeatRate":"60"
       }
    }

7.25.1 - snmptrap() source options

The snmptrap() driver has the following options. Only the filename() option is required, the others are optional.

filename()

Type:path
Default:

Description: The log file of snmptrapd. The AxoSyslog application reads the traps from this file.

In addition to traps, the log of snmptrapd may contain other messages (for example, daemon start/stop information, debug logs) as well. Currently AxoSyslog discards these messages.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

persist-name()

Type:string
Default:N/A

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

   Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

prefix()

Synopsis:prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

  • To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

  • To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

  • If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

Default value: .snmp. option.

set-message-macro()

Type:yes
Default:yes

Description: The snmptrap() source automatically parses the traps into name-value pairs, so you can handle the content of the trap as a structured message. Consequently, you might not even need the ${MESSAGE} part of the log message. If set-message-macro() is set to no, AxoSyslog leaves the ${MESSAGE} part empty. If set-message-macro() is set to yes, AxoSyslog generates a regular log message from the trap.

7.26 - sun-streams: Collect messages on Sun Solaris

Solaris uses its STREAMS framework to send messages to the syslogd process. Solaris 2.5.1 and above uses an IPC called door in addition to STREAMS, to confirm the delivery of a message. The AxoSyslog application supports the IPC mechanism via the door() option (see below).

The sun-streams() driver has a single required argument specifying the STREAMS device to open, and the door() option. For the list of available optional parameters, see sun-streams() source options.

Declaration:

   sun-streams(<name_of_the_streams_device> door(<filename_of_the_door>));

Example: Using the sun-streams() driver

   source s_stream {
        sun-streams("/dev/log" door("/etc/.syslog_door"));
    };

7.26.1 - sun-streams() source options

The sun-streams() driver has the following options.

door()

Type:string
Default:none

Description: Specifies the filename of a door to open, needed on Solaris above 2.5.1.

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

follow-freq()

Type:number
Default:1

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

keep-timestamp()

Type:yes or no
Default:yes

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Type:number
Default:100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Type:number
Default:100

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

log-msg-size()

Type:number (bytes)
Default:Use the global log-msg-size() option, which defaults to 65536 (64 KiB).

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

Type:string
Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

optional()

Type:yes or no
Default:

Description: Instruct AxoSyslog to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.

pad-size()

Type:number
Default:0

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

program-override()

Type:string
Default:

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Type:name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

7.27 - syslog: Collect messages using the IETF-syslog protocol

The syslog() driver can receive messages from the network using the standard IETF-syslog protocol (as described in RFC5424-26). UDP, TCP, and TLS-encrypted TCP can all be used to transport the messages.

For the list of available optional parameters, see syslog() source options.

Declaration:

   syslog(ip() port() transport() options());

Example: Using the syslog() driver

TCP source listening on the localhost on port 1999.

   source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); };

UDP source with defaults.

   source s_udp { syslog( transport("udp")); };

Encrypted source where the client is also authenticated. For details on the encryption settings, see TLS options.

   source s_syslog_tls{ syslog(
        ip(10.100.20.40)
        transport("tls")
        tls(
        peer-verify(required-trusted)
        ca-dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')
        key-file('/opt/syslog-ng/etc/syslog-ng/keys/server_privatekey.pem')
        cert-file('/opt/syslog-ng/etc/syslog-ng/keys/server_certificate.pem')
        )
        );
    };

7.27.1 - syslog() source options

The syslog() driver has the following options.

ca-dir()

Accepted values:Directory name
Default:none

Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.

This option can be used together with the optional ca-file() option.

ca-file()

Accepted values:File name
Default:empty

Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.

Example format in configuration:

   ca-file("/etc/pki/tls/certs/ca-bundle.crt")

dynamic-window-size()

Type:number
Default:0

Description: The size of the dynamic control window used during flow-control. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

ebpf()

Available in AxoSyslog version 4.2 and newer.

By default, the kernel chooses the receive socket for a specific UDP randomly based on the source IP/port of the sender. You can customize this algorithm using the Extended Berkeley Packet Filter (eBPF) plugin. The ebpf() option changes the kernel’s SO_REUSEPORT algorithm so that all messages are randomly placed into one of the UDP sockets. The decision which UDP socket buffer a datagram is placed is made for every datagram, and not once for every stream. This means that messages are perfectly load-balanced across your set of UDP sockets. While this resolves the imbalance between the sockets and results in perfect load balancing, you will lose ordering between messages from the same sender, which is the price to pay for increased throughput.

source s_network {
    network(
        transport("udp")
        so-reuseport(1) persist-name("udp1")
        ebpf(reuseport(sockets(4)))
    );
    network(transport("udp") so-reuseport(1) persist-name("udp2"));
    network(transport("udp") so-reuseport(1) persist-name("udp3"));
    network(transport("udp") so-reuseport(1) persist-name("udp4"));
};

For a detailed introduction, see the Scaling syslog to 1M EPS with eBPF blog post.

encoding()

Type:string
Default:

Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.

flags()

Type:assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8
Default:empty set

Description: Specifies the log parsing options of the source.

  • assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

    Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

    Example: using the no-header flag with the syslog-parser() parser

    The following example illustrates using the no-header flag with the syslog-parser() parser:

        parser p_syslog {
          syslog-parser(
            flags(no-header)
          );
        };
    
  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

        source s_dell {
            network(
                port(2000)
                flags(no-hostname)
            );
        };
    
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

  • no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

    If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

  • dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  • store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

    Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

    For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

    The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

  • threaded: The threaded flag enables multithreading for the source. For details on multithreading, see Multithreading and scaling.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Type:string
Default:N/A

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Type:string
Default:N/A

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

host-override()

Type:string
Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

interface()

Type:string
Default:None

Description: Bind to the specified interface instead of an IP address. Available in 3.19 and later.

ip() or localip()

Type:string
Default:0.0.0.0

Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.

If you specify a multicast bind address and use the udp transport, AxoSyslog automatically joins the necessary multicast group. TCP does not support multicasting.

ip-protocol()

Type:number
Default:4

Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).

Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).

For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.

   source s_network_tcp { network( transport("tcp") ip("::") ip-protocol(6) port(601) ); };

ip-tos()

Type:number
Default:0

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()

Type:number
Default:0

Description: Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Type:yes or no
Default:yes

Description: Specifies whether connections to sources should be closed when AxoSyslog is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the AxoSyslog connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep-hostname()

Type:yes or no
Default:no

Description: Enable or disable hostname rewriting.

  • If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.

  • If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

keep-timestamp()

Type:yes or no
Default:yes

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

listen-backlog()

Type:integer
Default:256

Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:

   source s_network {
        network(
            ip("192.168.1.1")
            transport("tcp")
            listen-backlog(2048)
            );
    };

log-fetch-limit()

Type:number
Default:100

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Type:number
Default:100

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

If the max-connections() option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection. For optimal performance when receiving messages from AxoSyslog clients, make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example: Initial window size of a connection

If log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log-msg-size()

Type:number (bytes)
Default:Use the global log-msg-size() option, which defaults to 65536 (64 KiB).

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

max-connections()

Type:number
Default:10

Description: Specifies the maximum number of simultaneous connections.

pad-size()

Type:number
Default:0

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

port() or localport()

Type:number
Default:

In case of TCP transport: 514

In case of UDP transport: 514

Description: The port number to bind to.

program-override()

Type:string
Default:

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

so-broadcast()

Type:yes or no
Default:no

Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.

so-keepalive()

Type:yes or no
Default:no

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-rcvbuf()

Type:number
Default:0

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so-reuseport()

Type:yes or no
Default:no

Description: Enables SO_REUSEPORT on systems that support it. When enabled, the kernel allows multiple UDP sockets to be bound to the same port, and the kernel load-balances incoming UDP datagrams to the sockets. The sockets are distributed based on the hash of (srcip, dstip, srcport, dstport), so the same listener should be receiving packets from the same endpoint. For example:

   source {
            network(transport("udp") so-reuseport(1) port(2000) persist-name("udp1"));
            network(so-reuseport(1) port(2000) persist-name("udp2"));
            network(so-reuseport(1) port(2000) persist-name("udp3"));
            network(so-reuseport(1) port(2000) persist-name("udp4"));
    };

Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-sndbuf()

Type:number
Default:0

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()

Type:string
Default:

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Type:name of the timezone, or the timezone offset
Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

transport()

Type:udp, tcp, tls, proxied-tcp, proxied-tls, proxied-tls-passthrough, text-with-nuls
Default:tcp

Description: Specifies the protocol used to receive messages from the source.

For detailed information about how AxoSyslog supports the proxied-tcp, the proxied-tls, and the proxied-tls-passthrough parameters, see Proxy Protocol support.

text-with-nuls: Allows embedded NUL characters in the message from a TCP source, that is, AxoSyslog will not delimiter the incoming messages on NUL characters, only on newline characters (contrary to tcp transport, which splits the incoming log on newline characters and NUL characters).

trim-large-messages()

Type:yes
Default:Use the global trim-large-messages() option, which defaults to no.

Description: Determines what AxoSyslog does with incoming log messages that are received using the IETF-syslog protocol using the syslog() driver, and are longer than the value of log-msg-size(). Other drivers ignore the trim-large-messages() option.

  • If set to no, AxoSyslog drops the incoming log message.

  • If set to yes, AxoSyslog trims the incoming log message to the size set in log-msg-size(), and adds the trimmed tag to the message. The rest of the message is dropped. You can use the tag to filter on such messages.

        filter f_trimmed {
            tags("trimmed");
        };
    

    If AxoSyslog trims a log message, it sends a debug-level log message to its internal() source.

    As a result of trimming, a parser could fail to parse the trimmed message. For example, a trimmed JSON or XML message will not be valid JSON or XML.

Available in AxoSyslog version 3.21 and later.

Uses the value of the global option if not specified.

tls()

Type:tls options
Default:n/a

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

use-dns()

Type:yes, no, persist_only
Default:yes

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example, from /etc/hosts). The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

use-fqdn()

Type:yes or no
Default:no

Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.

7.28 - syslog-ng-otlp(): Receive logs from another node using OpenTelemetry

Available in AxoSyslog version 4.4 and later.

The syslog-ng-otlp() source and destination allows you to transfer the internal representation of log messages between AxoSyslog instances using the OpenTelemetry protocol. Unlike the traditional syslog-ng() drivers that rely on simple TCP connections, syslog-ng-otlp() leverages the OpenTelemetry protocol for efficient and reliable log message transmission.

The key benefits of the syslog-ng-otlp() drivers are:

  • scalability (via the workers() option),
  • built-in application layer acknowledgement,
  • support for Google service authentication (ADC or ALTS), and
  • improved load balancing capabilities.

To use it, configure a syslog-ng-otlp() destination on the sender node, and a syslog-ng-otlp() source on the receiver node, like this:

destination d_syslog_ng_otlp {
  syslog-ng-otlp(url("your-receiver-syslog-ng-instance:4317"));
};
source s_syslog_ng_otlp {
  syslog-ng-otlp(port(4317));
};

Options

The syslog-ng-otlp() source has the following options.

auth()

You can set authentication in the auth() option of the driver. By default, authentication is disabled (auth(insecure())).

The following authentication methods are available in the auth() block:

adc()

Application Default Credentials (ADC). This authentication method is only available for destinations.

alts()

Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.

source {
    opentelemetry(
      port(4317)
      auth(alts())
    );
  };
destination {
    loki(
      port(12345)
      auth(alts())
    );
  };
source {
    syslog-ng-otlp(
      port(4317)
      auth(alts())
    );
  };

insecure()

This is the default method, authentication is disabled (auth(insecure())).

tls()

tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.

destination {
    opentelemetry(
      url("your-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
destination {
    loki(
      url("your-loki-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
destination {
    syslog-ng-otlp(
      url("your-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };

Note:

  • tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
  • The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.

chain-hostnames()

Accepted values:yes, no
Default:no

Description: Enable or disable the chained hostname format. For details, see the chain-hostnames() global option.

channel-args()

Type:See description
Default:-

Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:

    channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
    )

concurrent-requests()

Type:integer
Default:2

Description: Configures the maximal number of in-flight gRPC requests per worker. Setting this value in the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater than or equal to the number of clients, but this can increase the memory usage.

default-facility()

Type:facility string
Default:kern

Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.

default-level()

Type:string
Default:notice

Description: The default level value if the PRIORITY entry does not exist.

default-priority()

Type:priority string
Default:

Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).

dns-cache()

Accepted values:yes, no
Default:no

Description: Enable or disable DNS cache usage.

ebpf()

Available in AxoSyslog version 4.2 and newer.

By default, the kernel chooses the receive socket for a specific UDP randomly based on the source IP/port of the sender. You can customize this algorithm using the Extended Berkeley Packet Filter (eBPF) plugin. The ebpf() option changes the kernel’s SO_REUSEPORT algorithm so that all messages are randomly placed into one of the UDP sockets. The decision which UDP socket buffer a datagram is placed is made for every datagram, and not once for every stream. This means that messages are perfectly load-balanced across your set of UDP sockets. While this resolves the imbalance between the sockets and results in perfect load balancing, you will lose ordering between messages from the same sender, which is the price to pay for increased throughput.

source s_network {
    network(
        transport("udp")
        so-reuseport(1) persist-name("udp1")
        ebpf(reuseport(sockets(4)))
    );
    network(transport("udp") so-reuseport(1) persist-name("udp2"));
    network(transport("udp") so-reuseport(1) persist-name("udp3"));
    network(transp