Google BigQuery, macOS, and Windows XML support in syslog-ng version 4.6

Version 4.6 of syslog-ng gives you exciting new features to better integrate your telemetry pipeline to the services used by your organization.

• Packages are available for various platforms.
• Our AxoSyslog project provides cloud-ready container images and Helm charts.

We cover the highlights of the release in this post, for the in-depth details of every change, read the release notes on the GitHub Releases page. You can also find them in the AxoSyslog documentation.

Google BigQuery destination

The bigquery() destination feeds your data to Google BigQuery via the high-performance gRPC based API. BigQuery is a serverless and cost-effective enterprise data warehouse, often used to store PBs of log data for long time retention.

Example config:

bigquery(
    project("test-project")
    dataset("test-dataset")
    table("test-table")
    workers(8)

    schema(
        "message" => "$MESSAGE"
        "app" STRING => "$PROGRAM"
        "host" STRING => "$HOST"
        "pid" INTEGER => int("$PID")
    )

    on-error("drop-property")

    # or alternatively instead of schema():
    # protobuf-schema("/tmp/test.proto"
    #                 => "$MESSAGE", "$PROGRAM", "$HOST", "$PID")

    # keep-alive(time(20000) timeout(10000) max-pings-without-data(0))
);

• You can set the columns and their respective type and template with theschema() option. The available types are: STRING, BYTES, INTEGER,FLOAT, BOOLEAN, TIMESTAMP, DATE, TIME, DATETIME, JSON, NUMERIC, BIGNUMERIC, GEOGRAPHY, RECORD, INTERVAL.
• Alternatively you can import a .proto file with the protobuf-schema() option, and map the templates for each column.
• For details about the different parameters, see the AxoSyslog documentation.

Collecting local system logs on macOS

Two new sources have been added for macOS: darwin-oslog() and darwin-oslog-stream().  The darwin-oslog() source builds on the native OSLog framework, and replaces the earlier file-source based solution. The syslog-ng system() source automatically uses this new source on darwin platforms if the darwinosl plugin is available.

This plugin is available only on macOS 10.15 Catalina and above, the first version that has the OSLog API. For details, see our Enhance macOS logging with syslog-ng’s native macOS system() source blog post, and the AxoSyslog documentation.

Parsing Windows Eventlog XMLs

The new windows-eventlog-xml-parser() processes Windows Eventlog XMLs and turns them into a set of name-value pairs that is easy to represent as JSON. This parser makes it possible to turn the relatively difficult-to-use XML format of these messages into an easier to use JSON representation, that is also smaller in size.

Example configuration:

parser p_win {
    windows-eventlog-xml-parser(prefix(".winlog."));
};

For details, see the AxoSyslog documentation.

OpenTelemetry improvements

• The opentelemetry() source now supports multiple worker threads. This enables processing of the incoming OTLP messages on multiple threads, and results in higher throughput.
• To reduce bandwidth use, the opentelemetry() sources and destinations now support compression that can result in 1:10 compression ratios while transferring log data.

Other changes

• The cloud-auth() plugin now supports  user-managed-service-account() style authentication that allows the use of service accounts built into VMs running in GCP. You can read more on this here: https://cloud.google.com/compute/docs/access/authenticate-workloads#curl

Summary

This release takes syslog-ng another step closer to becoming a real cloud-native observability tool. We hope you’ll give it a try using the cloud-ready images of AxoSyslog, our syslog-ng distribution.

For the complete list of smaller changes and bug fixes, see the release notes. Stay tuned for more exciting features in the upcoming releases!

Thank you for everyone contributing with bug reports, feature requests, or pull requests. Feedback and any kind of contribution are always appreciated. Visit syslog-ng’s GitHub page or join Axoflow’s Discord server to reach out to us, or subscribe to the Axoflow newsletter to receive updates about syslog-ng and our observability and logging-related products.

For an overview on how our platform enhances syslog-ng based log collection with metrics, see the Metrics for syslog-ng based log management infrastructures blog post.