Version 4.9 of AxoSyslog, our syslog-ng™ fork (syslog-ng is a trademark of One Identity) brings you a destination for sending data to ClickHouse, various gRPC improvements, tons of new features for FilterX, and numerous bug fixes for issues reported in the AxoSyslog and the syslog-ng projects.

Packages are available for Debian and Ubuntu from our APT repository.
RPM packages are available in the Assets section of the GitHub Releases page. See our blog post for details on installing AxoSyslog on RHEL, AlmaLinux, or Fedora!
We also provide cloud-ready container images and Helm charts.
AxoSyslog 4.9 is a binary-compatible drop-in replacement for syslog-ng up to version 4.7.1 (and possibly newer).

This post covers the release’s highlights. For in-depth details of every change, see the release notes on the GitHub Releases page. You can also find the details of the new features in the AxoSyslog documentation.

Send data to ClickHouse

The new clickhouse() destination can send data to self-hosted ClickHouse databases using its gRPC interface (currently ClickHouse Cloud does not support
the gRPC interface).

For example, the following configuration (sends data to the default localhost:9100 URL):

destination {
  clickhouse(
    database("default")
    table("demo_table")
    user("your-username")
    password("your-password")
    schema(
      "user_id" UInt32 => $R_MSEC,
      "message" String => "$MSG",
      "timestamp" DateTime => "$R_UNIXTIME",
      "metric" Float32 => 3.14
    )
  );
};

For a detailed tutorial, see our Sending log data to ClickHouse with AxoSyslog blog post!

gRPC consolidation

We’ve consolidated our gRPC-based drivers, so now all these drivers have the common options of our gRPC implementation. As a result, the gRPC-based destinations:

Can use templates and macros in their header() values.

Now have the keep-alive() options time(), timeout(), and max-pings-without-data() :

opentelemetry(
    ...
    keep-alive(time(20000) timeout(10000) max-pings-without-data(0))
);

Have the frac-digits(), local-time-zone(), on-error(), send-time-zone(), template-escape(), time-zone(), ts-format() options. – The bigquery() destination now accepts the auth() option, and can use different authentication methods, like adc(), alts(), insecure() and tls(). For example:
```
bigquery (
    ...
    auth(
        tls(
            ca-file("/path/to/ca.pem")
            key-file("/path/to/key.pem")
            cert-file("/path/to/cert.pem")
        )
    )
);
```
The loki() destination now has the batch-bytes() and compression() options.

Log tapping

You can now attach to the standard IO of the syslog-ng process to check its output or its internal logs. For example:

# Display the syslog-ng output for ten seconds
$ syslog-ng-ctl attach stdio --seconds 10

# Change log level to trace and display trace level log messages for 10 seconds
$ syslog-ng-ctl attach logs --seconds 10 --log-level trace

FilterX

FilterX (AxoSyslog’s advanced parsing and filtering language) became a publicly available feature in AxoSyslog after the 4.8 release. If you aren’t familiar with FilterX, read our FilterX introduction blog for details. In 4.9. we’ve added several new features to FilterX, including the following highlights (for a full list, check the FilterX features) section of the release notes.

CEF and LEEF

FilterX now has parsers to process messages received in the Common Event Format (CEF) and Log Event Extended Format (LEEF) formats. For LEEF, both v1 and v2 messages are supported.

For example, the following LEEF-formatted message includes mandatory and custom (extension) fields:

LEEF:1.0|Microsoft|MSExchange|4.0 SP1|15345|src=192.0.2.0 dst=172.50.123.1 sev=5cat=anomaly srcPort=81 dstPort=21 usrName=john.smith

The following FilterX expression parses it and converts it into JSON format:

filterx {
    ${PARSED_MESSAGE} = json(parse_leef(${MESSAGE}));
};

The content of the JSON object for this message will be:

{
"version":"1.0",
"vendor":"Microsoft",
"product_name":"MSExchange",
"product_version":"4.0 SP1",
"event_id":"15345",
"extensions": {
    "src":"192.0.2.0",
    "dst":"172.50.123.1",
    "sev":"5cat=anomaly",
    "srcPort":"81",
    "dstPort":"21",
    "usrName":"john.smith"
    }
}

Windows Event Logs and XML

With FilterX you can parse XML objects, and a specialized parser handles Windows Event Logs. Given the following input:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
    <System>
        <Provider Name='EventCreate'/>
        <EventID Qualifiers='0'>999</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime='2024-01-12T09:30:12.1566754Z'/>
        <EventRecordID>934</EventRecordID>
        <Correlation/>
        <Execution ProcessID='0' ThreadID='0'/>
        <Channel>Application</Channel>
        <Computer>DESKTOP-2MBFIV7</Computer>
        <Security UserID='S-1-5-21-3714454296-2738353472-899133108-1001'/>
    </System>
    <RenderingInfo Culture='en-US'>
        <Message>foobar</Message>
        <Level>Error</Level>
        <Task></Task>
        <Opcode>Info</Opcode>
        <Channel></Channel>
        <Provider></Provider>
        <Keywords>
            <Keyword>Classic</Keyword>
        </Keywords>
    </RenderingInfo>
    <EventData>
        <Data Name='param1'>foo</Data>
        <Data Name='param2'>bar</Data>
    </EventData>
</Event>

The parse_windows_eventlog_xml() parser creates the following JSON object:

{
    "Event": {
        "@xmlns": "http://schemas.microsoft.com/win/2004/08/events/event",
        "System": {
            "Provider": {"@Name": "EventCreate"},
            "EventID": {"@Qualifiers": "0", "#text": "999"},
            "Version": "0",
            "Level": "2",
            "Task": "0",
            "Opcode": "0",
            "Keywords": "0x80000000000000",
            "TimeCreated": {"@SystemTime": "2024-01-12T09:30:12.1566754Z"},
            "EventRecordID": "934",
            "Correlation": "",
            "Execution": {"@ProcessID": "0", "@ThreadID": "0"},
            "Channel": "Application",
            "Computer": "DESKTOP-2MBFIV7",
            "Security": {"@UserID": "S-1-5-21-3714454296-2738353472-899133108-1001"},
        },
        "RenderingInfo": {
            "@Culture": "en-US",
            "Message": "foobar",
            "Level": "Error",
            "Task": "",
            "Opcode": "Info",
            "Channel": "",
            "Provider": "",
            "Keywords": {"Keyword": "Classic"},
        },
        "EventData": {
            "Data": {
                "param1": "foo",
                "param2": "bar",
            },
        },
    },
}

Metrics for FilterX

You can now use the update_metric() function to increment metrics, similarly to metrics-probe parser. For example:

update_metric("filterx_metric", labels={"msg": $MSG, "foo": "foovalue"}, level=1, increment=$INCREMENT);

String matches

You can use the startswith, endswith, and includes functions to check prefix, suffix, and substring matches without having to use complex (and computationally costly) regular expressions.

Other changes

For the complete list of changes, see the release notes on the GitHub Releases page. You can also find the details of the new features in the AxoSyslog documentation.

Summary

This is the first official release supporting FilterX! We hope you’ll give it a try, and are really excited to hear your feedback about it. For the complete list of smaller changes and bug fixes that we haven’t covered here, see the release notes. Stay tuned for more exciting features in the upcoming releases!

Thank you for everyone contributing with bug reports, feature requests, or pull requests. Feedback and any contribution is always appreciated. Visit AxoSyslog GitHub page or join Axoflow’s Discord server to reach out to us.

On-deman Webinar