Collect native macOS system logs

Starting with version 4.6.0, AxoSyslog can collect logs on macOS using its native OSLog framework using the darwin-oslog() and darwin-oslog-stream() source drivers.

• darwin-oslog(): This source builds on the native OSLog framework, and replaces the earlier file-source based solution.
• darwin-oslog-stream(): Provides a live log stream feed.

darwin-oslog()

This source is based on the native OSLog Framework to read logs from the local store of the unified logging system on darwin OSes. The AxoSyslog system() source automatically uses this new source on darwin platforms if the darwinosl plugin is available. This plugin is available only on macOS 10.15 Catalina and above, the first version that has the OSLog API.

Note that the persistent OSLog store usually keeps about 7 days of logs on disk.

The darwin-oslog() source has the following options:

chain-hostnames()

Description: Enable or disable the chained hostname format. For details, see the chain-hostnames() global option.

default-facility()

Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.

default-level()

Description: The default level value if the PRIORITY entry does not exist.

default-priority()

Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).

default-severity()

Description: Assigns a default severity to messages received from this source when the message doesn’t specify one. Accepts the standard syslog severity keywords (for example, emerg, alert, crit, err, warning, notice, info, debug). Combined with default-facility() to compute the message priority.

–>

dns-cache()

Description: Enable or disable DNS cache usage.

do-not-use-bookmark()

Description: By default, AxoSyslog continues to read the logs from the last remembered position after a restart. If this option is set to yes, it will always start reading from the end or beginning of the available log list (depending on the setting of the go-reverse() option).

fetch-delay()

Description: Controls the time AxoSyslog waits between reading and sending log messages. This is a fraction of a second, where wait_time = 1 second / n, so n=1 means that only about 1 log is sent in each second, and n=1000000 means only 1 microsecond is the delay between read/write attempts. The maximal value of this parameter is 1000000. Note that increasing the value of this parameter (thus lowering delay time) can increase log feed performance, but at the same time could increase system load.

fetch-retry-delay()

Description: Controls how many seconds AxoSyslog waits before trying to check for new logs if there were no more logs to read the last time.

filter-predicate()

Description: String for native macOS log message filtering using predicates. For example, the following predicate selects AirDrop logs: subsystem=="com.apple.sharing" and category=="AirDrop"

flags()

Description: Specifies the log parsing options of the source. For example:

source { network(flags(check-hostname, check-program)); };

assume-utf8

The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

check-hostname

When receiving messages, AxoSyslog can check whether the hostname contains valid characters.

Valid characters are:

• alphanumeric characters (A-Z, a-z, 0-9)
• the dash (-) and underscore (_) characters
• the dot (.) and the colon (:) characters
• the @ and slash (/)

If the hostname contains invalid characters, AxoSyslog sets the syslog.invalid_hostname tag for the message, and doesn’t parse the ${HOST} field from the message.

check-program

Available in version 4.10 and later.

If the check-program flag is enabled, AxoSyslog validates the ${PROGRAM} field for RFC3164-formatted messages. Valid program names meet the following criteria:

• Contain only these characters: [a-zA-Z0-9-_/().]
• Include at least one alphabetical character.

If the program name fails validation, it’s considered to be part of the log message.

You can also enable this behavior globally using the check-program global option.

dont-store-legacy-msghdr

By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

empty-lines

Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

exit-on-eof

If this flag is set on a source, AxoSyslog stops when an EOF (end of file) is received. Available in version 4.9 and later.

expect-hostname

If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

guess-timezone

Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

kernel

The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

no-header

The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

The following example illustrates using the no-header flag with the syslog-parser() parser:

parser p_syslog {
    syslog-parser(
    flags(no-header)
    );
};

no-hostname

Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

source s_dell {
    network(
        port(2000)
        flags(no-hostname)
    );
};

no-multi-line

The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

no-parse

By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

sanitize-utf8

When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

store-legacy-msghdr

By default, AxoSyslog stores the original incoming header of the log message, so this flag is active. To disable it, use the dont-store-legacy-msghdr flag.

store-raw-message

Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

syslog-protocol

The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

validate-utf8

The validate-utf8 flag enables encoding-verification for messages.

Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

go-reverse()

Description: Set it to yes to process the logs in a reverse order (from latest to oldest).

hook-commands()

Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.

Using hook-commands() when AxoSyslog starts or stops

To execute an external program when AxoSyslog starts or stops, use the following options:

startup()

Description: Defines the external program that is executed as AxoSyslog starts.

shutdown()

Description: Defines the external program that is executed as AxoSyslog stops.

Using the hook-commands() when AxoSyslog reloads

To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:

setup()

Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.

teardown()

Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.

Example: Using hook-commands() with a network source

In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.

The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.

source {
    network(transport(udp)
    hook-commands(
          startup("iptables -I LOGCHAIN 1 -p udp --dport 514 -j ACCEPT")
          shutdown("iptables -D LOGCHAIN 1")
        )
     );
};

host-override()

Description: Replaces the ${HOST} part of the message with the parameter string.

internal()

Description: Marks this pipeline element as internal. Elements marked as internal() are treated as an implementation detail, so for example statistics of the given pipe are available only on higher stats level. This option is mainly useful for developers or when writing SCL blocks and integrations.

keep-hostname()

Description: Enable or disable hostname rewriting.

• If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.

• If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

keep-timestamp()

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Description: Specifies the source window size - the maximum number of in-flight messages permitted by the source before flow control is enforced. This only applies when flow-control is enabled.

<p>If you change the value of <code>log-iw-size()</code> and <code>keep-alive()</code> is enabled, the change will affect only new connections, the <code>log-iw-size()</code> of kept-alive connections will not change. To apply the new <code>log-iw-size()</code> value to every connection, <a href="/docs/axosyslog-core/quickstart/managing-and-checking-linux/#restart-axosyslog">restart the <code>syslog-ng</code> service</a>. A simple configuration reload is <em>NOT</em> sufficient.</p>

Note that when using disk-buffer(), the messages stored on disk are not included in the window size calculation. For details about the effects of this parameter, see Managing incoming and outgoing messages with flow-control.

log-prefix() (DEPRECATED)

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

long-hostnames()

Obsolete alias for chain-hostnames().

–>

max-bookmark-distance()

Description: The maximum distance in seconds that a bookmark can point backwards. That is, if AxoSyslog was stopped for 10 minutes and max-bookmark-distance() is set to 60, then AxoSyslog will start reading the logs from 60 seconds before the startup, losing 9 minutes of logs.

normalize-hostnames()

Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.

persist-name()

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

program-override()

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

read-old-records()

Description: If set to yes, AxoSyslog starts reading logs from the oldest available log when it’s first started on a system, or if there are no bookmarks for some reason

sdata-prefix()

Available in AxoSyslog 4.1 and later.

Description: Adds a specific string before the names of the parsed SDATA fields to store the name-value pairs created from the SDATA fields separately. Note that unless the value of sdata-prefix starts with .SDATA., using this option excludes the parsed fields from the sdata and rfc5424 scopes of the value pairs.

tags()

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

time-zone()

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

use-dns()

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example, from /etc/hosts). The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

use-fqdn()

Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.

use-syslogng-pid()

Description: When enabled, AxoSyslog overrides the PID from the original message with the PID of the syslog-ng process.

darwin-oslog-stream()

This source is a wrapper around the OS command line log stream command that provides a live log stream feed. Unlike the darwin-oslog() source, the live stream can contain non-persistent log events as well. This might result in a large number of log events every second.

This source has only one option.

params()

Description: A string that can contain all the possible params the macOS log tool can accept. The source uses the “–style” internally (defaults to ndjson), so use templates or rewrite rules to format the final output. Use the def-osl-stream-params string to reference the default values when extending them with your own.

For a full reference, see the output of the log --help stream and man log commands.