source: Read, receive, and collect log messages

• 1: How sources work
• 2: Arr logs
• 3: Collect native macOS system logs
• 4: default-network-drivers: Receive and parse common syslog messages
• 4.1: default-network-drivers() source options
• 5: internal: Collect internal messages
• 5.1: internal() source options
• 6: file: Collect messages from text files
• 6.1: Notes on reading kernel messages
• 6.2: file() source options
• 7: wildcard-file: Collect messages from multiple text files
• 7.1: wildcard-file() source options
• 8: Hypr Audit Trail and Hypr App Audit Trail
• 9: Jellyfin logs
• 10: linux-audit: Collect messages from Linux audit logs
• 10.1: linux-audit() source options
• 11: kubernetes: Collect and parse the Kubernetes CRI (Container Runtime Interface) format
• 11.1: kubernetes() source options
• 12: mbox: Convert local email messages to log messages
• 12.1: mbox() source options
• 13: mqtt: receiving messages from an MQTT broker
• 13.1: Prerequisites to using the mqtt() source
• 13.2: Limitations to using the mqtt() source
• 13.3: Options of the mqtt() source
• 14: network: Collect messages using the RFC3164 protocol (network() driver)
• 14.1: network() source options
• 14.2: Proxy Protocol support
• 14.2.1: The working mechanism behind the Proxy Protocol
• 14.2.2: Proxy Protocol: configuration and output examples
• 15: nodejs: Receive JSON messages from nodejs applications
• 15.1: nodejs() source options
• 16: osquery: Collect and parse osquery result logs
• 16.1: osquery() source options
• 17: Receive logs, metrics, and traces from OpenTelemetry
• 18: pacct: Collect process accounting logs on Linux
• 18.1: pacct() options
• 19: Pi-hole Faster Than Light logs
• 20: pipe: Collect messages from named pipes
• 20.1: pipe() source options
• 21: program: Receive messages from external applications
• 21.1: program() source options
• 22: python: writing server-style Python sources
• 22.1: Python LogMessage API
• 22.2: python() and python-fetcher() source options
• 23: python-fetcher: writing fetcher-style Python sources
• 24: qBittorrent logs
• 25: snmptrap: Read Net-SNMP traps
• 25.1: snmptrap() source options
• 26: sun-streams: Collect messages on Sun Solaris
• 26.1: sun-streams() source options
• 27: syslog: Collect messages using the IETF-syslog protocol
• 27.1: syslog() source options
• 28: syslog-ng-otlp(): Receive logs from another node using OpenTelemetry
• 29: system: Collect the system-specific log messages of a platform
• 29.1: system() source options
• 30: systemd-journal: Collect messages from the systemd-journal system log storage
• 30.1: systemd-journal() source options
• 31: systemd-syslog: Collect systemd messages using a socket
• 31.1: systemd-syslog() source options
• 32: tcp, tcp6, udp, udp6: OBSOLETE - Collect messages from remote hosts using the BSD syslog protocol
• 32.1: tcp(), tcp6(), udp() and udp6() source options: OBSOLETE
• 32.1.1: Change an old source driver to the network() driver
• 33: unix-stream, unix-dgram: Collect messages from UNIX domain sockets
• 33.1: UNIX credentials and other metadata
• 33.2: unix-stream() and unix-dgram() source options
• 34: stdin: Collect messages from the standard input stream
• 34.1: stdin() source options

1 - How sources work

A source is where AxoSyslog receives log messages. Sources consist of one or more drivers, each defining where and how messages are received.

To define a source, add a source statement to the syslog-ng.conf configuration file using the following syntax:

   source <identifier> {
        source-driver(params); source-driver(params); ...
    };

Example: A simple source statement

The following source statement receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address.

   source s_demo_tcp {
        network(ip(10.1.2.3) port(1999));
    };

Example: A source statement using two source drivers

The following source statement receives messages on the 1999 TCP port and the 1999 UDP port of the interface having the 10.1.2.3 IP address.

   source s_demo_two_drivers {
        network(ip(10.1.2.3) port(1999));
        network(ip(10.1.2.3) port(1999) transport("udp"));
    };

Example: Setting default priority and facility

If the message received by the source does not have a proper syslog header, you can use the default-facility() and default-priority() options to set the facility and priority of the messages. Note that these values are applied only to messages that do not set these parameters in their header.

   source headerless_messages { network(default-facility(syslog) default-priority(emerg)); };

Define a source only once. The same source can be used in several log paths. Duplicating sources causes AxoSyslog to open the source (TCP/IP port, file, and so on) more than once, which might cause problems. For example, include the /dev/log file source only in one source statement, and use this statement in more than one log path if needed.

To collect log messages on a specific platform, it is important to know how the native syslogd communicates on that platform. The following table summarizes the operation methods of syslogd on some of the tested platforms:

Communication methods used between the applications and syslogd

Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using the SOCK_STREAM style — as used under Linux — is called unix-stream.

Example: Source statement on a Linux based operating system

The following source statement collects the following log messages:

• internal(): Messages generated by syslog-ng.

• network(transport(“udp”)): Messages arriving to the 514/UDP port of any interface of the host.

• unix-dgram("/dev/log");: Messages arriving to the /dev/log socket.

   source s_demo {
        internal();
        network(transport("udp"));
        unix-dgram("/dev/log");
    };

Sources list

The following table lists the source drivers available in syslog-ng.

Source drivers available in syslog-ng

2 - Arr logs

Starting with version 4.7.0, AxoSyslog can collect logs of the Lidarr, Prowlarr, Radarr, Readarr, and Sonarr (often referred to as “*Arr” or “*Arrs”) applications.

Use the new *arr() sources to read various *arr logs:

• lidarr()
• prowlarr()
• radarr()
• readarr()
• sonarr()
• whisparr()

Example minimal config:

source s_radarr {
    radarr(
    dir("/path/to/my/radarr/log/dir")
    );
};

The logging module is stored in the <prefix><module> name-value pair, for example: .radarr.module => ImportListSyncService.

You can modify the prefix with the prefix() option.

This driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

3 - Collect native macOS system logs

Starting with version 4.6.0, AxoSyslog can collect logs on macOS using its native OSLog framework using the darwin-oslog() and darwin-oslog-stream() source drivers.

• darwin-oslog(): This source builds on the native OSLog framework, and replaces the earlier file-source based solution.
• darwin-oslog-stream(): Provides a live log stream feed.

darwin-oslog()

This source is based on the native OSLog Framework to read logs from the local store of the unified logging system on darwin OSes. The AxoSyslog system() source automatically uses this new source on darwin platforms if the darwinosl plugin is available. This plugin is available only on macOS 10.15 Catalina and above, the first version that has the OSLog API.

Note that the persistent OSLog store usually keeps about 7 days of logs on disk.

The darwin-oslog() source has the following options:

filter-predicate()

Description: String for native macOS log message filtering using predicates. For example, the following predicate selects AirDrop logs: subsystem=="com.apple.sharing" and category=="AirDrop"

do-not-use-bookmark()

Description: By default, AxoSyslog continues to read the logs from the last remembered position after a restart. If this option is set to yes, it will always start reading from the end or beginning of the available log list (depending on the setting of the go-reverse() option).

fetch-delay()

Description: Controls the time AxoSyslog waits between reading and sending log messages. This is a fraction of a second, where wait_time = 1 second / n, so n=1 means that only about 1 log is sent in each second, and n=1000000 means only 1 microsecond is the delay between read/write attempts. The maximal value of this parameter is 1000000. Note that increasing the value of this parameter (thus lowering delay time) can increase log feed performance, but at the same time could increase system load.

fetch-retry-delay()

Description: Controls how many seconds AxoSyslog waits before trying to check for new logs if there were no more logs to read the last time.

go-reverse()

Description: Set it to yes to process the logs in a reverse order (from latest to oldest).

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

max-bookmark-distance()

Description: The maximum distance in seconds that a bookmark can point backwards. That is, if AxoSyslog was stopped for 10 minutes and max-bookmark-distance() is set to 60, then AxoSyslog will start reading the logs from 60 seconds before the startup, losing 9 minutes of logs.

read-old-records()

Description: If set to yes, AxoSyslog starts reading logs from the oldest available log when it’s first started on a system, or if there are no bookmarks for some reason

darwin-oslog-stream()

This source is a wrapper around the OS command line log stream command that provides a live log stream feed. Unlike the darwin-oslog() source, the live stream can contain non-persistent log events as well. This might result in a large number of log events every second.

This source has only one option.

params()

Description: A string that can contain all the possible params the macOS log tool can accept. The source uses the “–style” internally (defaults to ndjson), so use templates or rewrite rules to format the final output. Use the def-osl-stream-params string to reference the default values when extending them with your own.

For a full reference, see the output of the log --help stream and man log commands.

4 - default-network-drivers: Receive and parse common syslog messages

The default-network-drivers() source is a special source that uses multiple source drivers to receive and parse several different types of syslog messages from the network. Available in version 3.16 and later.

To use the default-network-drivers() source, the scl.conf file must be included in your AxoSyslog configuration:

   @include "scl.conf"

Also, make sure that your SELinux, AppArmor, and firewall settings permit AxoSyslog to access the ports where you want to receive messages, and that no other application is using these ports. By default, the default-network-drivers() source accepts messages on the following ports:

• 514, both TCP and UDP, for RFC3164 (BSD-syslog) formatted traffic
• 601 TCP, for RFC5424 (IETF-syslog) formatted traffic
• 6514 TCP, for TLS-encrypted traffic

In addition to receiving messages on different ports and in different formats, this source tries to parse the messages automatically. If successful, it sets the ${.app.name} name-value pair to the name of the application that sent the log message. Currently it uses the following procedures.

Parsing RFC3164-formatted messages

For RFC3164-formatted messages (that is, messages received on the ports set in options udp-port() and tcp-port() which default to port 514), AxoSyslog attempts to use the following parsers. If a parser cannot parse the message, it passes the original message to the next parser.

1. Parse the incoming raw message as a message from a Cisco device.

2. Parse the incoming message as an RFC3164-formatted message.

   • If the incoming message was sent by a AxoSyslog client using the syslog-ng() destination, parse its fields as a AxoSyslog message.

     The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving AxoSyslog component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first AxoSyslog component in your infrastructure has received it, which is important if you want to forward a message for example, to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.

   • Otherwise, apply the application adapters if the message was sent from an application that already has a specific parser in AxoSyslog (for example, Splunk Common Information Model (CIM), iptables, or sudo).

Parsing RFC5424-formatted messages

For RFC5424-formatted messages (that is, messages received on the ports set in options rfc5424-tls-port() and rfc5424-tcp-port(), which default to port 601 and 6514), AxoSyslog parses the message according to RFC5424, then attempts apply the application adapters if the message was sent from an application that already has a specific parser in AxoSyslog (for example, Splunk Common Information Model (CIM), iptables, or sudo).

Example: Using the default-network-drivers() driver

The following example uses only the default settings.

   source s_network {
        default-network-drivers();
    };

The following example can receive TLS-encrypted connections on the default port (port 6514).

   source s_network {
        default-network-drivers(
            tls(
                key-file("/path/to/ssl-private-key")
                cert-file("/path/to/ssl-cert")
            )
        );
    };

4.1 - default-network-drivers() source options

The systemd-journal() driver has the following options.

ca-dir()

Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.

This option can be used together with the optional ca-file() option.

ca-file()

Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.

Example format in configuration:

   ca-file("/etc/pki/tls/certs/ca-bundle.crt")

flags()

Description: Specifies the log parsing options of the source.

• assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

• empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

• expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

• guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

• kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

• no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

  Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

  Note

  Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.

  Example: using the no-header flag with the syslog-parser() parser

  The following example illustrates using the no-header flag with the syslog-parser() parser:

      parser p_syslog {
        syslog-parser(
          flags(no-header)
        );
      };
  

• no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

      source s_dell {
          network(
              port(2000)
              flags(no-hostname)
          );
      };
  

• no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

• no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

  If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

• dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

• sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

• store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

• syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

• validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

  The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

log-msg-size()

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

max-connections()

Description: Specifies the maximum number of simultaneous connections.

Note that the total number of connections the default-network-drivers() source can use is 3*max-connections(), because this value applies to the network(tcp), syslog(tcp), and syslog(tls) connections individually.

rfc5424-tcp-port()

Description: The TCP port number where the default-network-drivers() source receives RFC5424-formatted (IETF-syslog) messages.

rfc5424-tls-port()

Description: The TCP port number where the default-network-drivers() source receives RFC5424-formatted (IETF-syslog), TLS-encrypted messages.

   source s_network {
        default-network-drivers(
            tls(
                key-file("/path/to/ssl-private-key")
                cert-file("/path/to/ssl-cert")
            )
        );
    };

tcp-port()

Description: The TCP port number where the default-network-drivers() source receives RFC3164-formatted (BSD-syslog) messages.

tls()

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

udp-port()

Description: The UDP port number where the default-network-drivers() source receives RFC3164-formatted (BSD-syslog) messages.

5 - internal: Collect internal messages

All messages generated internally by AxoSyslog use this special source. To collect warnings, errors and notices from AxoSyslog itself, include this source in one of your source statements.

   internal()

The AxoSyslog application will issue a warning upon startup if none of the defined log paths reference this driver.

Example: Using the internal() driver

   source s_local { internal(); };

The AxoSyslog application sends the following message types from the internal() source:

• fatal: Priority value: critical (2), Facility value: syslog (5)

• error: Priority value: error (3), Facility value: syslog (5)

• warning: Priority value: warning (4), Facility value: syslog (5)

• notice: Priority value: notice (5), Facility value: syslog (5)

• info: Priority value: info (6), Facility value: syslog (5)

5.1 - internal() source options

The internal() driver has the following options:

host-override()

Description: Replaces the ${HOST} part of the message with the parameter string.

log-iw-size()

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

normalize-hostnames()

Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.

program-override()

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

tags()

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

use-fqdn()

Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

6 - file: Collect messages from text files

Collects log messages from plain-text files, for example, from the logfiles of an Apache webserver. If you want to use wildcards in the filename, use the wildcard-file() source.

The AxoSyslog application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When AxoSyslog is restarted, it records the position of the last sent log message in the /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist file, and continues to send messages from this position after the restart.

The file driver has a single required parameter specifying the file to open. If you want to use wildcards in the filename, use the wildcard-file() source. For the list of available optional parameters, see file() source options.

Declaration:

   file("filename");

Example: Using the file() driver

   source s_file {
        file("/var/log/messages");
    };

Example: Tailing files

The following source checks the access.log file every second for new messages.

   source s_tail {
        file("/var/log/apache/access.log" follow-freq(1) flags(no-parse));
    };

6.1 - Notes on reading kernel messages

Note the following points when reading kernel messages on various platforms.

• The kernel usually sends log messages to a special file (/dev/kmsg on BSDs, /proc/kmsg on Linux). The file() driver reads log messages from such files. The AxoSyslog application can periodically check the file for new log messages if the follow-freq() option is set.

• On Linux, the klogd daemon can be used in addition to AxoSyslog to read kernel messages and forward them to syslog-ng. klogd used to preprocess kernel messages to resolve symbols and so on, but as this is deprecated by ksymoops there is really no point in running both klogd and AxoSyslog in parallel. Also note that running two processes reading /proc/kmsg at the same time might result in dead-locks.

• When using AxoSyslog to read messages from the /proc/kmsg file, AxoSyslog automatically disables the follow-freq() parameter to avoid blocking the file.

• To read the kernel messages on HP-UX platforms, use the following options in the source statement:

      file("/dev/klog" program-override("kernel") flags(kernel) follow-freq(0));
  

6.2 - file() source options

The file() driver has the following options:

default-facility()

Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.

default-priority()

Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).

encoding()

Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.

flags()

Description: Specifies the log parsing options of the source.

• assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

• empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

• expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

• guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

• kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

• no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

  Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

  Note

  Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.

  Example: using the no-header flag with the syslog-parser() parser

  The following example illustrates using the no-header flag with the syslog-parser() parser:

      parser p_syslog {
        syslog-parser(
          flags(no-header)
        );
      };
  

• no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

      source s_dell {
          network(
              port(2000)
              flags(no-hostname)
          );
      };
  

• no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

• no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

  If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

• dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

• sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

• store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

• syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

• validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

  The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

follow-freq()

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

keep-timestamp()

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Description: The size of the initial window, this value is used during flow control. Make sure that log-iw-size() is larger than the value of log-fetch-limit().

log-msg-size()

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

multi-line-garbage()

Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.

To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.

multi-line-mode()

Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:

• indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.

      source s_tomcat {
          file("/var/log/tomcat/xxx.log" multi-line-mode(indented));
      };
  

• prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.

• prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.

  The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.

• smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.

  smart mode is available in AxoSyslog version 4.2 and newer.

  The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.

multi-line-prefix()

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.

Example: Processing Tomcat logs

The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

   2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    SEVERE: Catalina.start:
    LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use null:8080
           at org.apache.catalina.connector.Connector.start(Connector.java:1138)
           at org.apache.catalina.core.StandardService.start(StandardService.java:531)
           at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
           at org.apache.catalina.startup.Catalina.start(Catalina.java:583)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)
    2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    INFO: Server startup in 1206 ms
    2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-8080
    2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
    INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:

source s_file{file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0) multi-line-mode(regexp) multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));};
    };

Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.

multi-line-suffix()

Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.

To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.

pad-size()

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

program-override()

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

tags()

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

7 - wildcard-file: Collect messages from multiple text files

The wildcard-file() source collects log messages from multiple plain-text files from multiple directories. The wildcard-file() source is available in AxoSyslog version 3.10 and later.

The AxoSyslog application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When AxoSyslog is restarted, it records the position of the last sent log message in the persist file, and continues to send messages from this position after the restart. The location of the persist file depends on the package you installed AxoSyslog from, typically it is /var/lib/syslog-ng/syslog-ng.persist or /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist.

Declaration:

   wildcard-file(
        base-dir("<pathname>")
        filename-pattern("<filename>")
    );

Note the following important points:

• You can use the * and ? wildcard characters in the filename (the filename-pattern() option), but not in the path (the base-dir() option).

• If you use multiple wildcard-file() sources in your configuration, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.

• When using wildcards, AxoSyslog monitors every matching file (up to the limit set in the max-files() option), and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because AxoSyslog uses the inotify feature of the kernel. Set the max-files() option at least to the number of files you want to monitor. If the wildcard-file source matches more files than the value of the max-files() option, it is random which files will AxoSyslog actually monitor. The default value of max-files() is 100.

• If the message does not have a proper syslog header, AxoSyslog treats messages received from files as sent by the user facility. Use the default-facility() and default-priority() options in the source definition to assign a different facility if needed.
• For every message that AxoSyslog reads from the source files, the path and name of the file is available in the ${FILE_NAME} macro.

Required parameters: base-dir(), filename-pattern(). For the list of available optional parameters, see wildcard-file() source options.

Example: Using the wildcard-file() driver

The following example monitors every file with the .log extension in the /var/log directory for log messages.

   source s_files {
        wildcard-file(
            base-dir("/var/log")
            filename-pattern("*.log")
            recursive(no)
            follow-freq(1)
        );
    };

7.1 - wildcard-file() source options

The wildcard-file() driver has the following options:

base-dir()

Description: The path to the directory that contains the log files to monitor, for example, base-dir("/var/log"). To monitor also the subdirectories of the base directory, use the recursive(yes) option. For details, see recursive().

   source s_files {
        wildcard-file(
            base-dir("/var/log")
            filename-pattern("*.log")
            recursive(no)
            follow-freq(1)
        );
    };

default-facility()

Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.

default-priority()

Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).

encoding()

Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.

filename-pattern()

Description: The filename to read messages from, without the path. You can use the * and ? wildcard characters, without regular expression and character range support. You cannot use the * and ? literally in the pattern.

For example, filename-pattern("*.log") matches the syslog.log and auth.log files, but does not match the access_log file. The filename-pattern("*log") pattern matches all three.

• * matches an arbitrary string, including an empty string
• ? matches an arbitrary character

   source s_files {
        wildcard-file(
            base-dir("/var/log")
            filename-pattern("*.log")
            recursive(no)
            follow-freq(1)
        );
    };

flags()

Description: Specifies the log parsing options of the source.

• assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

• empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

• expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

• guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

• kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

• no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

  Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

  Note

  Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.

  Example: using the no-header flag with the syslog-parser() parser

  The following example illustrates using the no-header flag with the syslog-parser() parser:

      parser p_syslog {
        syslog-parser(
          flags(no-header)
        );
      };
  

• no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

      source s_dell {
          network(
              port(2000)
              flags(no-hostname)
          );
      };
  

• no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

• no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

  If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

• dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

• sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

• store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

• syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

• validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

  The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

follow-freq()

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

keep-timestamp()

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Description: The size of the initial window, this value is used during flow control. Make sure that log-iw-size() is larger than the value of log-fetch-limit().

When using wildcards in the filenames, AxoSyslog attempts to read log-fetch-limit() number of messages from each file. For optimal performance, make sure that log-iw-size() is greater than log-fetch-limit()*max-files(). Note that to avoid performance problems, if log-iw-size()/max-files() is smaller than 100, AxoSyslog automatically sets log-iw-size() to max-files()*100.

Example: Initial window size of file sources

If log-fetch-limit() is 100, and your wildcard file source has 200 files, then log-iw-size() should be at least 20000.

log-msg-size()

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

max-files()

Description: Limits the number of files that the wildcard-file source monitors.

When using wildcards, AxoSyslog monitors every matching file (up to the limit set in the max-files() option), and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because AxoSyslog uses the inotify feature of the kernel. Set the max-files() option at least to the number of files you want to monitor. If the wildcard-file source matches more files than the value of the max-files() option, it is random which files will AxoSyslog actually monitor. The default value of max-files() is 100.

monitor-method()

Description: If the platform supports inotify, AxoSyslog uses it automatically to detect changes to the source files. If inotify is not available, AxoSyslog polls the files as set in the follow-freq() option. To force AxoSyslog poll the files even if inotify is available, set this option to poll.

multi-line-garbage()

Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.

To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.

multi-line-mode()

Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:

• indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.

      source s_tomcat {
          file("/var/log/tomcat/xxx.log" multi-line-mode(indented));
      };
  

• prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.

• prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.

  The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.

• smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.

  smart mode is available in AxoSyslog version 4.2 and newer.

  The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.

multi-line-prefix()

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.

Example: Processing Tomcat logs

The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

   2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    SEVERE: Catalina.start:
    LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use null:8080
           at org.apache.catalina.connector.Connector.start(Connector.java:1138)
           at org.apache.catalina.core.StandardService.start(StandardService.java:531)
           at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
           at org.apache.catalina.startup.Catalina.start(Catalina.java:583)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)
    2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    INFO: Server startup in 1206 ms
    2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-8080
    2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
    INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:

source s_file{file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0) multi-line-mode(regexp) multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));};
    };

Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.

multi-line-suffix()

Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.

To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.

pad-size()

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

program-override()

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

recursive()

Description: When enabled, AxoSyslog monitors every subdirectory of the path set in the base-dir() option, and reads log messages from files with matching filenames. The recursive option can be used together with wildcards in the filename.

Example: Monitoring multiple directories

The following example reads files having the .log extension from the /var/log/ directory and its subdirectories, including for example, the /var/log/apt/history.log file.

   source s_file_subdirectories {
        wildcard-file(
            base-dir("/var/log")
            filename-pattern("*.log")
            recursive(yes)
            follow-freq(1)
            log-fetch-limit(100)
        );
    };

tags()

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

8 - Hypr Audit Trail and Hypr App Audit Trail

Starting with version 4.2.0, AxoSyslog can fetch events from the Hypr REST API using the hypr-audit-trail() and hypr-app-audit-trail() source drivers.

• hypr-audit-trail(): is a source driver that pulls messages from the Hypr API, associated to any RP Application ID.
• hypr-app-audit-trail(): is a source driver that pulls messages from the Hypr API, but only those associated to a specific RP Application ID.

Hypr Audit Trail

The hypr-audit-trail() source queries the Hypr API for the list of potential applications at startup, then monitors the audit trail for each of the detected applications.

Note: Applications that are registered after syslog-ng is started are not recognized.

To start following those audit trails, you must restart syslog-ng.

Example minimal configuration:

source s_hypr {
    hypr-audit-trail(
        url('https://<custom domain>.hypr.com')
        bearer-token('<base64 encoded bearer token>')
    );
};

A more detailed example:

source s_hypr {
    hypr-audit-trail(
        url('https://<custom domain>.hypr.com')
        bearer-token('<base64 encoded bearer token>')
        page-size(<number of results to return in a single page>)
        initial-hours(<number of hours to search backward on initial fetch>)
        application-skip-list('HYPRDefaultApplication', 'HYPRDefaultWorkstationApplication')
        log-level('INFO')
        flags(<optional flags passed to the source>)
        ignore-persistence(<yes/no>)
    );
};

Available options:

• url(): custom URL for Hypr API access ('https://<custom domain>.hypr.com')
• bearer-token(): base64 encoded authentication token from Hypr
• page-size(): number of results to return in a single page (optional - defaults to 100)
• initial-hours(): number of hours to search backward on initial fetch (optional - defaults to 4)
• application-skip-list(): list of rpAppIds not to retrieve from Hypr (optional - defaults to 'HYPRDefaultApplication', 'HYPRDefaultWorkstationApplication')
• log-level(): logging level, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" (optional - defaults to "INFO")
• flags(): flags passed to the source, can be used for example to disable message parsing with flags(no-parse) (optional - defaults to empty)
• ignore-persistence(): ignores the saved value in the persist file, and starts querying from the current time (optional - defaults to no)

Hypr App Audit Trail

The hypr-app-audit-trail() monitors the audit trail for one specific RP Application ID. This driver requires the rp-app-id() parameter in order to operate.

Available options:

• url(): custom URL for Hypr API access ('https://<custom domain>.hypr.com')
• bearer-token(): base64 encoded authentication token from Hypr
• rp-app-id(): the RP Application ID for the application to monitor
• page-size(): number of results to return in a single page (optional - defaults to 100)
• initial-hours(): number of hours to search backward on initial fetch (optional - defaults to 4)
• log-level(): logging level, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" (optional - defaults to "INFO")
• flags(): flags passed to the source, can be used for example to disable message parsing with flags(no-parse) (optional - defaults to empty)
• ignore-persistence(): ignores the saved value in the persist file, and starts querying from the current time (optional - defaults to no)

Acknowledgements

This documentation page is based on the README file of the hypr-audit-trail() source, written by Dan Elder.

9 - Jellyfin logs

Starting with version 4.7.0, you can use the jellyfin() source to read Jellyfin logs from its log file output.

Example minimal configuration:

source s_jellyfin {
    jellyfin(
    base-dir("/path/to/my/jellyfin/root/log/dir")
    filename-pattern("log_*.log")
    );
};

For more details about Jellyfin logging, see:

• https://jellyfin.org/docs/general/administration/configuration/#main-configuration
• https://jellyfin.org/docs/general/administration/configuration/#log-directory

As the jellyfin() source is based on the wildcard-file() source, you can use the wildcard-file() source options.

This driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

10 - linux-audit: Collect messages from Linux audit logs

It reads and automatically parses the Linux audit logs. You can override the file name using the filename() parameter and the prefix for the created name-value pairs using the prefix() parameter. Any additional parameters are passed to the file source.

Declaration:

   linux-audit(options);

Example: Using the linux-audit() driver

source s_auditd {
    linux-audit(
        prefix("test.")
        hook-commands(
            startup("auditctl -w /etc/ -p wa")
            shutdown("auditctl -W /etc/ -p wa")
        )
    );
};
            

10.1 - linux-audit() source options

The file() driver has the following options:

filename()

Description: The log file of linux-audit. The AxoSyslog application reads the Linux audit logs from this file.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. Note that if you use an empty prefix (prefix("")) or one starting with a dot, AxoSyslog might replace the original value of an existing macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

11 - kubernetes: Collect and parse the Kubernetes CRI (Container Runtime Interface) format

The kubernetes() source collects container logs managed by the Kubelet. It reads plain-text and JSON-formatted container logs (as described in the Container Runtime Interface (CRI) design proposal), for example, from the /var/log/containers or /var/log/pods files, and enriches them with various metadata retrieved from the Kubernetes API.

The kubernetes() source is available in AxoSyslog version 3.37 and later.

By default, it reads the /var/log/containers folder and extracts:

• the log content, and
• Kubernetes metadata, for example, namespace, pod, and container information.

The Kubernetes-related metadata is available in name-value pairs with the .k8s. prefix. The following table shows the retrieved metadata and their source:

Declaration

   kubernetes(
        base-dir("<pathname>")
    );

11.1 - kubernetes() source options

The kubernetes() source has the following options:

base-dir()

Description: The path to the directory that contains the log files, for example, base-dir("/var/log/pods").

cluster-name()

Description: The name of the Kubernetes cluster.

key-delimiter()

Description: The delimiter character to use when parsing flattened keys. Supports Only single characters.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

The prefix() option is optional and its default value is ".k8s.".

12 - mbox: Convert local email messages to log messages

Using the mbox() driver, AxoSyslog can read email messages from local mbox files, and convert them to multiline log messages.

This driver has only one required option, the filename of the mbox file. To use the mbox() driver, the scl.conf file must be included in your AxoSyslog configuration:

   @include "scl.conf"

The mbox() driver is actually a reusable configuration snippet configured to read log messages using the file() driver. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the configuration snippet on GitHub.

Example: Using the mbox() driver

The following example reads the emails of the root user on the AxoSyslog host.

   @include "scl.conf"
    source root-mbox {
        mbox("/var/spool/mail/root");
    };

12.1 - mbox() source options

The mbox() driver has the following option.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

13 - mqtt: receiving messages from an MQTT broker

From AxoSyslog version 3.35, you can use the mqtt() source to fetch messages from MQTT brokers.

The mqtt() source builds on the MQTT protocol, and uses its client and broker entities.

Declaration:

   source s_mqtt{
        mqtt(
            address("tcp://<hostname>:<port-number>")
            topic("<topic-name>")
        );
    };

Starting with AxoSyslog version 4.7, mqtt() source automatically sets the ${MQTT_TOPIC} name-value pair for the messages it receives. This is useful when the name of the topic contains MQTT wildcards ($, +, #). For example:

log {
    source { mqtt(topic("#")); };
    destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); };
};

Example: Using the mqtt() source in your configuration

The following example illustrates an mqtt() source configured to fetch messages from the MQTT broker running on localhost:4444 using the test/test topic, and send them to the localhost:4445 address.

    @version: current
    @include "scl.conf"
    source s_mqtt {
        mqtt(
            address("tcp://localhost:4444")
            topic("test/test")
        );
    };
    destination d_network {
        network(
            "localhost"
            port(4445)
        );
    };
    log {
        source(s_mqtt);
        destination(d_network);;
    };

13.1 - Prerequisites to using the mqtt() source

Using the current implementation of the mqtt() source has the following prerequisites:

• Installing the eclipse-paho-mqtt-c library.

  Note

  The default package manager for some Linux operating systems contains the eclipse-paho-mqtt-c library, but depending on your OS, you may have to install the library manually. For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.

• Having a broker entity in a functional MQTT system.

  Note

  In your configuration, you will specify the broker entity of your MQTT system in the address() option of your mqtt() source.

13.2 - Limitations to using the mqtt() source

Using the mqtt() source of AxoSyslog has the following limitations:

• You can only use the mqtt() source with AxoSyslog version 3.35 or higher.
• You cannot use the mqtt() source without installing the the eclipse-paho-mqtt-c library.

• For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.

• The current implementation of the mqtt() source supports versions 3.1 and 3.1.1 of the MQTT protocol

13.3 - Options of the mqtt() source

The mqtt() source has the following options.

Required options: address(), fallback-topic(), and topic().

address()

Description: Specifies the hostname or IP address, and the port number of the MQTT broker to which AxoSyslog will send the log messages.

Syntax: <protocol type>://<host>:<port>

client-id()

Description: The client-id() is used to identify the client to the MQTT server, which stores session data for each client. The session data can contains information regarding which message has been sent, received. It is not possible to set the client-id() to an empty string. To always start a new session see the cleansession() option.

cleansession()

Description: This option instruments the MQTT broker to clean the session data when connecting. The session data contains information about which message was processed.

http-proxy()

Description: Specifies HTTP/HTTPS proxy for WebSocket connections.

keep-alive()

Description: Specifies the number of seconds that AxoSyslog keeps the connection between the broker and clients open in case there is no message traffic. When keep-alive() number of seconds pass, the connection is terminated, and you have to reconnect.

On the MQTT side, the keep alive function provides a workaround method to access connections that are still open.

password()

Description: The password used to authenticate on the MQTT broker.

persist-name()

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

   Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

qos()

Description: The Quality of Service (QoS) level in MQTT messaging is an agreement between sender and receiver on the guarantee of delivering a message.

tls()

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

The following options are relevant for the mqtt() tls() block: ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), ssl-version(), use-system-cert-store().

topic()

Description: Required option. Specifies the MQTT topic.

username()

Description: The username used to authenticate on the MQTT broker.

14 - network: Collect messages using the RFC3164 protocol (network() driver)

The network() source driver can receive syslog messages conforming to RFC3164 from the network using the TCP, TLS, and UDP networking protocols.

• UDP is a simple datagram oriented protocol, which provides “best effort service” to transfer messages between hosts. It may lose messages, and no attempt is made to retransmit lost messages. The BSD-syslog protocol traditionally uses UDP.

  Use UDP only if you have no other choice.

• TCP provides connection-oriented service: the client and the server establish a connection, each message is acknowledged, and lost packets are resent. TCP can detect lost connections, and messages are lost, only if the TCP connection breaks. When a TCP connection is broken, messages that the client has sent but were not yet received on the server are lost.

• The AxoSyslog application supports TLS (Transport Layer Security, also known as SSL) over TCP. For details, see Encrypting log messages with TLS.

Declaration:

   network([options]);

By default, the network() driver binds to 0.0.0.0, meaning that it listens on every available IPV4 interface on the TCP/514 port. To limit accepted connections to only one interface, use the localip() parameter. To listen on IPv6 addresses, use the ip-protocol(6) option.

Example: Using the network() driver

Using only the default settings: listen on every available IPV4 interface on the TCP/514 port.

   source s_network {
        network();
    };

UDP source listening on 192.168.1.1 (the default port for UDP is 514):

   source s_network {
        network(
            ip("192.168.1.1")
            transport("udp")
        );
    };

TCP source listening on the IPv6 localhost, port 2222:

   source s_network6 {
        network(
            ip("::1")
            transport("tcp")
            port(2222)
            ip-protocol(6)
        );
    };

A TCP source listening on a TLS-encrypted channel.

   source s_network {
        network(
            transport("tls")
            port(2222)
            tls(peer-verify("required-trusted")
                key-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.key")
                cert-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt")
            );
        );
    };

A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslog messages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses both the IETF-syslog message format and the protocol. For details, see syslog: Collect messages using the IETF-syslog protocol.

   source s_tcp_syslog {
        network(
            ip("127.0.0.1")
            flags(syslog-protocol)
        );
    };

For details on the options of the network() source, see network() source options.

14.1 - network() source options

The network() driver has the following options.

ca-dir()

Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.

This option can be used together with the optional ca-file() option.

ca-file()

Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.

Example format in configuration:

   ca-file("/etc/pki/tls/certs/ca-bundle.crt")

dynamic-window-size()

Description: The size of the dynamic control window used during flow-control. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

ebpf()

Available in AxoSyslog version 4.2 and newer.

By default, the kernel chooses the receive socket for a specific UDP randomly based on the source IP/port of the sender. You can customize this algorithm using the Extended Berkeley Packet Filter (eBPF) plugin. The ebpf() option changes the kernel’s SO_REUSEPORT algorithm so that all messages are randomly placed into one of the UDP sockets. The decision which UDP socket buffer a datagram is placed is made for every datagram, and not once for every stream. This means that messages are perfectly load-balanced across your set of UDP sockets. While this resolves the imbalance between the sockets and results in perfect load balancing, you will lose ordering between messages from the same sender, which is the price to pay for increased throughput.

source s_network {
    network(
        transport("udp")
        so-reuseport(1) persist-name("udp1")
        ebpf(reuseport(sockets(4)))
    );
    network(transport("udp") so-reuseport(1) persist-name("udp2"));
    network(transport("udp") so-reuseport(1) persist-name("udp3"));
    network(transport("udp") so-reuseport(1) persist-name("udp4"));
};

For a detailed introduction, see the Scaling syslog to 1M EPS with eBPF blog post.

encoding()

Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.

flags()

Description: Specifies the log parsing options of the source.

• assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

• empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

• expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

• guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

• kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

• no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

  Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

  Note

  Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.

  Example: using the no-header flag with the syslog-parser() parser

  The following example illustrates using the no-header flag with the syslog-parser() parser:

      parser p_syslog {
        syslog-parser(
          flags(no-header)
        );
      };
  

• no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

      source s_dell {
          network(
              port(2000)
              flags(no-hostname)
          );
      };
  

• no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

• no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

  If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

• dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

• sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

• store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

• syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

• validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

  The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

• threaded: The threaded flag enables multithreading for the source. For details on multithreading, see Multithreading and scaling.

  Note

  The syslog source uses multiple threads only if the source uses the tls or tcp transport protocols.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

host-override()

Description: Replaces the ${HOST} part of the message with the parameter string.

interface()

Description: Bind to the specified interface instead of an IP address. Available in 3.19 and later.

ip() or localip()

Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.

If you specify a multicast bind address and use the udp transport, AxoSyslog automatically joins the necessary multicast group. TCP does not support multicasting.

ip-protocol()

Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).

Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).

For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.

   source s_network_tcp { network( transport("tcp") ip("::") ip-protocol(6) port(601) ); };

ip-tos()

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()

Description: Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Description: Specifies whether connections to sources should be closed when AxoSyslog is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the AxoSyslog connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep-hostname()

Description: Enable or disable hostname rewriting.

• If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.

• If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

keep-timestamp()

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

listen-backlog()

Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:

   source s_network {
        network(
            ip("192.168.1.1")
            transport("tcp")
            listen-backlog(2048)
            );
    };

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.

If the max-connections() option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection. For optimal performance when receiving messages from AxoSyslog clients, make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example: Initial window size of a connection

If log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log-msg-size()

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

max-connections()

Description: Specifies the maximum number of simultaneous connections.

pad-size()

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:

   Padding was set, and couldn't read enough bytes

port() or localport()

Description: The port number to bind to.

program-override()

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

so-broadcast()

Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.

so-keepalive()

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-rcvbuf()

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so-reuseport()

Description: Enables SO_REUSEPORT on systems that support it. When enabled, the kernel allows multiple UDP sockets to be bound to the same port, and the kernel load-balances incoming UDP datagrams to the sockets. The sockets are distributed based on the hash of (srcip, dstip, srcport, dstport), so the same listener should be receiving packets from the same endpoint. For example:

   source {
            network(transport("udp") so-reuseport(1) port(2000) persist-name("udp1"));
            network(so-reuseport(1) port(2000) persist-name("udp2"));
            network(so-reuseport(1) port(2000) persist-name("udp3"));
            network(so-reuseport(1) port(2000) persist-name("udp4"));
    };

Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-sndbuf()

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

transport()

Description: Specifies the protocol used to receive messages from the source.

For detailed information about how AxoSyslog supports the proxied-tcp, the proxied-tls, and the proxied-tls-passthrough parameters, see Proxy Protocol support.

text-with-nuls: Allows embedded NUL characters in the message from a TCP source, that is, AxoSyslog will not delimiter the incoming messages on NUL characters, only on newline characters (contrary to tcp transport, which splits the incoming log on newline characters and NUL characters).

trim-large-messages()

Description: Determines what AxoSyslog does with incoming log messages that are received using the IETF-syslog protocol using the syslog() driver, and are longer than the value of log-msg-size(). Other drivers ignore the trim-large-messages() option.

• If set to no, AxoSyslog drops the incoming log message.

• If set to yes, AxoSyslog trims the incoming log message to the size set in log-msg-size(), and adds the trimmed tag to the message. The rest of the message is dropped. You can use the tag to filter on such messages.

      filter f_trimmed {
          tags("trimmed");
      };
  

  If AxoSyslog trims a log message, it sends a debug-level log message to its internal() source.

  As a result of trimming, a parser could fail to parse the trimmed message. For example, a trimmed JSON or XML message will not be valid JSON or XML.

Available in AxoSyslog version 3.21 and later.

Uses the value of the global option if not specified.

tls()

Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.

use-dns()

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example, from /etc/hosts). The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

use-fqdn()

Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.

14.2 - Proxy Protocol support

If you connect load balancers to your AxoSyslog application, AxoSyslog identifies every connection that is connected to the load balancers identically by default, regardless of the source IP or the source protocol. Essentially, the load balancer masks the source IP unless you enable Proxy Protocol support for your proxy TLS transport() to inject information about the original connection into the forwarded TCP session.

For further details about the working mechanism behind the Proxy Protocol support on AxoSyslog and the configuration details, see the following sections:

14.2.1 - The working mechanism behind the Proxy Protocol

This section describes how AxoSyslog supports the Proxy Protocol.

The working mechanism behind the Proxy Protocol

When using the Proxy Protocol during load balancing, AxoSyslog detects the information behind connections connected to the load balancer, then parses the injected information and adds the following macros to every message the comes through the connection later on:

• PROXY_SRCIP (the source IP of the proxy)

• PROXY_SRCPORT (the source port of the proxy)

• PROXY_DSTIP (the destination IP of the proxy)

• PROXY_DSTPORT (the destination port of the proxy)

14.2.2 - Proxy Protocol: configuration and output examples

This section provides information about enabling Proxy Protocol support in your network() source options, and an example configuration and output to illustrate how the Proxy Protocol method works in AxoSyslog.

For more information about the working mechanism of the Proxy Protocol, see The working mechanism behind the Proxy Protocol.

Enabling Proxy Protocol support for your network() source options

Unless you enable Proxy Protocol support for your network() source, AxoSyslog identifies every connection that is connected to the load balancers identically by default, regardless of the source IP or the source protocol.

To enable Proxy Protocol for your network() source, set the transport() parameter of your network() source to proxied-tcp or proxied-tls-passthrough, depending on your preference and configuration.

proxied-tls can be used in complex MITM (man in the middle) configurations, where the proxy header is sent encrypted within the same TLS session as the proxied messages.

When you enable Proxy Protocol support for your network() source, you can use the following configuration example with your AxoSyslog application.

Configuration

The following code sample illustrates how you can use the Proxy Protocol in your AxoSyslog configuration (using the transport() parameter set to proxied-tls-passthrough).

   @version: 3.35
    
    source s_tcp_pp {
      network (
        port(6666)
        transport("proxied-tls-passthrough")
        tls(
            key-file("/certs/certs/server/server.rsa")
            cert-file("/certs/certs/server/server.crt")
            ca-dir("/certs/certs/CA")
            peer-verify("required-trusted")
        )
      );
    };
    
    destination d_file {
      file("/var/log/proxy-proto.log" template("$(format-json --scope nv-pairs)\n"));
    };
    
    log {
      source(s_tcp_pp);
      destination(d_file);
    };

With this configuration, the Proxy Protocol method will perform injecting the information of the original connection into the forwarded TCP session, based on the working mechanism described in The working mechanism behind the Proxy Protocol.

The following example illustrates how the parsed macros will appear in the output.

Example: Output for the PROXY TCP4 192.168.1.1 10.10.0.1 1111 2222 input header

With the PROXY TCP4 192.168.1.1 10.10.0.1 1111 2222 input header, the output looks like this:

   {"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"1111","PROXIED_SRCIP":"192.168.1.1","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"2222","PROXIED_DSTIP":"10.10.0.1","PROGRAM":"TestMsg","MESSAGE":"","LEGACY_MSGHDR":"TestMsg","HOST_FROM":"localhost","HOST":"localhost"}

Note that the macros that AxoSyslog adds to the message appear in the output.

15 - nodejs: Receive JSON messages from nodejs applications

Using the nodejs() driver, AxoSyslog can receive application logs directly from nodejs applications that use the widespread Winston logging API. The AxoSyslog application automatically adds the .nodejs.winston. prefix to the name of the fields the extracted from the message.

To use the nodejs() driver, the scl.conf file must be included in your AxoSyslog configuration:

   @include "scl.conf"

The nodejs() driver is actually a reusable configuration snippet configured to receive log messages using the network() driver, and process its JSON contents. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the nodejs configuration snippet on GitHub.

Example: Using the nodejs() driver

The following example uses the default settings of the driver, listening for messages on port 9003 of every IP address of the AxoSyslog host.

   @include "scl.conf"
    source apps { nodejs(); };

The following example listens only on IP address 192.168.1.1, port 9999.

   @include "scl.conf"
    source apps {
        nodejs(
            localip(192.168.1.1)
            port(9999)
        )
    };

15.1 - nodejs() source options

The nodejs() driver has the following options.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

ip() or localip()

Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.

If you specify a multicast bind address and use the udp transport, AxoSyslog automatically joins the necessary multicast group. TCP does not support multicasting.

port() or localport()

Description: The port number to bind to.

16 - osquery: Collect and parse osquery result logs

The osquery application allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. You can make queries on demand, and also schedule them to run regularly.

The osquery() source of AxoSyslog allows you read the results of periodical osquery queries (from the /var/log/osquery/osqueryd.results.log file) and automatically parse the messages (if you want to use AxoSyslog to send log messages to osquery, read this blogpost). For example, you can:

• Create filters from the fields of the messages.

• Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on).

• Send the messages to a central location, for example, to Elasticsearch, directly from AxoSyslog.

The AxoSyslog application automatically adds the .osquery. prefix to the name of the fields the extracted from the message.

The osquery() source is available in AxoSyslog version 3.10 and later.

Prerequisites:

• To use the osquery() driver, the scl.conf file must be included in your AxoSyslog configuration:

      @include "scl.conf"
  

• AxoSyslog must be compiled with JSON-support enabled.

The osquery() driver is actually a reusable configuration snippet configured to read the osquery log file using the file() driver, and process its JSON contents. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

Example: Using the osquery() driver with the default settings

The following AxoSyslog configuration sample uses the default settings of the driver, reading osquery result logs from the /var/log/osquery/osqueryd.results.log file, and writes the log messages generated from the traps into a file.

   @version: 3.10
    @include "scl.conf"
    source s_osquery {
        osquery();
    };
    log {
        source(s_osquery);
        destination {
            file("/var/log/example.log");
        };
    };

Filter for messages related to loading Linux kernel modules:

   @version: 3.10
    @include "scl.conf"
    source s_osquery {
        osquery();
    };
    log {
        source(s_osquery);
        filter f_modules {
            "${.osquery.name}" eq "pack_incident-response_kernel_modules"
        };
        destination {
            file("/var/log/example.log");
        };
    };

Example: Using the osquery() driver with custom configuration

The following AxoSyslog configuration sample reads osquery result logs from the /tmp/osquery_input.log file, and writes the log messages generated from the traps into a file. Using the format-json template, the outgoing message will be a well-formed JSON message.

Input message:

   {"name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","calendarTime":"Fri Jul 21 10:04:41 2017 UTC","unixTime":"1500631481","decorations":{"host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","username":"myuser"},"columns":{"build_distro":"xenial","build_platform":"ubuntu","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","config_valid":"1","counter":"4","extensions":"active","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","pid":"19764","resident_size":"26416000","start_time":"1500629552","system_time":"223","user_time":"476","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","version":"2.5.0","watcher":"19762"},"action":"added"}

AxoSyslog configuration:

   @version: 3.10
    @include "scl.conf"
    source s_osquery {
        osquery(
          file(/tmp/osquery_input.log)
          prefix(.osquery.)
        );
    };
    destination d_file {
        file(
          "/tmp/output.txt"
          template("$(format_json --key .osquery.*)\n")
        );
    };
    log {
      source(s_osquery);
      destination(d_file);
      flags(flow-control);
    };

Outgoing message:

   Outgoing message; message='{"_osquery":{"unixTime":"1500631481","name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","user_time":"476","system_time":"223","start_time":"1500629552","resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","extensions":"active","counter":"4","config_valid":"1","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu","build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a'

To configure a destination to send the log messages to Elasticsearch, see elasticsearch2: DEPRECATED - Send messages directly to Elasticsearch version 2.0 or higher. For other destinations, see destination: Forward, send, and store log messages.

16.1 - osquery() source options

The osquery() driver has the following options.

file()

Description: The log file of osquery that stores the results of periodic queries. The AxoSyslog application reads the messages from this file.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

Default value:

.osquery. option.

17 - Receive logs, metrics, and traces from OpenTelemetry

Starting with version 4.3.0, AxoSyslog can receive logs, metrics, and traces from OpenTelemetry clients over the OpenTelemetry Protocol (OTLP/gRPC).

Example: Receiving OpenTelemetry data

The following example receives OpenTelemetry data and forwards it to an OpenTelemetry receiver. Note that by default, AxoSyslog doesn’t parse the fields of the incoming messages into name-value pairs, but are only available for forwarding using the opentelemetry() destination. To parse the fields into name-value pairs, use the opentelemetry() parser.

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(4317)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

auth()

You can set authentication in the auth() option of the driver. By default, authentication is disabled (auth(insecure())).

The following authentication methods are available in the auth() block:

adc()

Application Default Credentials (ADC). This authentication method is only available for destinations.

alts()

Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.

• Driver:
• opentelemetry()
• loki()
• syslog-ng-otlp()

source {
    opentelemetry(
      port(4317)
      auth(alts())
    );
  };

destination {
    loki(
      port(12345)
      auth(alts())
    );
  };

source {
    syslog-ng-otlp(
      port(4317)
      auth(alts())
    );
  };

insecure()

This is the default method, authentication is disabled (auth(insecure())).

tls()

tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.

• Driver:
• opentelemetry()
• loki()
• syslog-ng-otlp()

destination {
    opentelemetry(
      url("your-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };

destination {
    loki(
      url("your-loki-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };

destination {
    syslog-ng-otlp(
      url("your-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };

Note:

• tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
• The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.

channel-args()

Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:

    channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
    )

concurrent-requests()

Description: Configures the maximal number of in-flight gRPC requests per worker. Setting this value in the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater than or equal to the number of clients, but this can increase the memory usage.

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

port()

The port number to receive incoming connections. Default value: 4317

workers()

Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to process the messages of the source. Increasing the number of worker threads can drastically improve the performance of the source.

18 - pacct: Collect process accounting logs on Linux

Starting with version 3.2, AxoSyslog can collect process accounting logs on Linux systems.Process accounting is the method of recording and summarizing commands executed on Linux, for example, the commands being run, the user who executed the command, CPU time used by the process, exit code, and so on. When process accounting (also called pacct) is enabled on a system, the kernel writes accounting records to the /var/log/account/pacct file (might vary between different Linux distributions).

To use the pacct() driver, the following conditions must be met:

• The AxoSyslog application must be compiled with the --enable-pacct option. Execute the syslog-ng -V command to check if your binary supports process accounting.

• The pacctformat plugin must be loaded. By default, AxoSyslog automatically loads the available modules.

• The scl.conf file must be included in your AxoSyslog configuration:

      @include "scl.conf"
  

• Process accounting must be running on the host. You can enable it with the accton command.

The pacct() driver parses the fields of the accounting logs and transforms them into name-value pairs. The fields are defined in the manual page of the accounting log file (man acct), AxoSyslog prepends every field with the .pacct. prefix. For example, the ac_uid field that contains the id of the user who started the process will be available under the $.pacct.ac_uid name. These can be used as macros in templates, in filters to select specific messages, and so on.

To use the pacct() driver, use the following syntax.

   @version: 4.5.0
    @include "scl.conf"
    source s_pacct { pacct(); };
    ...
    log { source(s_pacct); destination(...); };

The pacct() driver is actually a reusable configuration snippet configured to handle Linux accounting logs. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the pacct configuration snippet on GitHub.

18.1 - pacct() options

The pacct() driver has the following options:

file()

Description: The file where the process accounting logs are stored — AxoSyslog reads accounting messages from this file.

follow-freq()

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

19 - Pi-hole Faster Than Light logs

Starting with version 4.6.0, AxoSyslog can collect logs of the Pi-hole FTL (Faster Than Light) application.

source s_pihole_ftl {
  pihole-ftl();
};

By default, the source reads the /var/log/pihole/FTL.log file. If the root directory of your Pi-hole installation is different, specify the directory where the FTL.log file is with the dir() option. You can find the root log directory by selecting Tools > Pi-hole diagnosis in the Pi-hole application. Otherwise, the pihole-ftl() source has the same parameters as the file() source.

The pihole-ftl() driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

20 - pipe: Collect messages from named pipes

The pipe driver opens a named pipe with the specified name and listens for messages. It is used as the native message delivery protocol on HP-UX.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. For the list of available optional parameters, see pipe() source options.

Declaration:

   pipe(filename);

Pipe is very similar to the file() driver, but there are a few differences, for example, pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg.

Example: Using the pipe() driver

   source s_pipe {
        pipe("/dev/pipe" pad-size(2048));
    };

20.1 - pipe() source options

The pipe driver has the following options:

create-dirs()

Description: Enable creating non-existing directories when creating files or socket files.

flags()

Description: Specifies the log parsing options of the source.

• assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

• empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

• expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

• guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

• kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

• no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

  Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

  Note

  Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.

  Example: using the no-header flag with the syslog-parser() parser

  The following example illustrates using the no-header flag with the syslog-parser() parser:

      parser p_syslog {
        syslog-parser(
          flags(no-header)
        );
      };
  

• no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

      source s_dell {
          network(
              port(2000)
              flags(no-hostname)
          );
      };
  

• no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

• no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

  If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

• dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

• sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

• store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

• syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

• validate-utf8: The validate-utf8 flag enables encoding-verification for messages.

  Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

  For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

  The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

follow-freq()

Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using the hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

   source {
       network(transport(udp)
        hook-commands(
              startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
              shutdown("iptables -D LOGCHAIN 1")
            )
         );
    };

keep-timestamp()

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()