Exploring the Exciting New Features of syslog-ng 4.1

The team at Axoflow, including our CEO, Balazs Scheidler, has been working tirelessly on bringing syslog-ng closer to the cloud-native ecosystem. This release is the first step in adapting syslog-ng for the modern observability supply chain. You might want to keep a close eye on these release blogs as interesting new features are yet to come.

Packages are available for various platforms. Detailed release notes can be found on the GitHub Releases page. Let’s go through some notable new features.

Metrics overhaul

Observability tools traditionally provide insights into how applications operate to make service outages or security incidents easier to diagnose and prevent. Enterprise-wide observability systems that take care of logs, metrics, and traces on behalf of a large set of applications, are complex applications in their own right, and we at Axoflow believe that as such, they need observability features built in.

syslog-ng has been often used as a component in such enterprise systems, but its observability features are lacking: although statistics have been available for a long time to give quantifiable information of an instance’s inner state, there are still big gaps in the provided metrics, and its format (CSV) does not meet modern standards. You find a brief introduction to the new metrics-related features in this post but stay tuned for the upcoming deep-dive post.

Note that these features are experimental and may change in the following syslog-ng releases.

Prometheus metrics format

A new metric system has been introduced to syslog-ng, where metrics are identified by names and partitioned by labels, similarly to the Prometheus data model.

Querying metrics samples with syslog-ng-ctl stats prometheus provides an output like this:

syslogng_events_allocated_bytes 18446744073709550560
syslogng_filtered_events_total{id="#anon-filter0",result="matched"} 0
syslogng_filtered_events_total{id="#anon-filter0",result="not_matched"} 2
syslogng_filtered_events_total{id="ff",result="matched"} 0
syslogng_filtered_events_total{id="ff",result="not_matched"} 2
syslogng_input_events_total{id="#anon-source0#0",driver_instance="-",result="processed"} 0
syslogng_input_events_total{id="s_network#1",result="processed"} 2
syslogng_internal_source{result="dropped"} 0
syslogng_internal_source{result="queued"} 0
syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="delivered"} 0
syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="dropped"} 0
syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="queued"} 0
syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="delivered"} 0
syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="dropped"} 0
syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="queued"} 0
syslogng_parsed_events_total{id="#anon-parser0",result="discarded"} 0
syslogng_parsed_events_total{id="#anon-parser0",result="processed"} 0
syslogng_scratch_buffers_bytes 0
syslogng_scratch_buffers_total 3
syslogng_tagged_events_total{id=".source.#anon-source0",result="processed"} 0
syslogng_tagged_events_total{id=".source.s_network",result="processed"} 2

Classification (metadata-based metrics)

metrics-probe(), a new parser counts messages passing through, based on the metadata of each message. The parser creates labeled metrics based on the specific fields of the event. Place it after your log path classifies and parses the messages for useful insights about the different types of events going through the pipeline, for example:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

For details, see the documentation of AxoSyslog, the cloud-native syslog-ng distribution.

Named log paths (path ingress/egress metrics)

Although the source and destination-focused metrics are useful for measuring the load and throughput of different connectors of syslog-ng, users are often interested in a specific log paths’ metrics. With syslog-ng 4.1, you can create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Ingress and egress metrics are available for named log paths:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

PROXY protocol v2

We have added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol used by network load balancers, such as Amazon Elastic Load Balancer and HAProxy, to carry original source/destination address information, as described in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Notable mentions

• date-parser() improvements, new format-date() template function.
• kubernetes() source now supports "." (dots) in field names, error logging is improved.
• Several disk-buffer() related fixes.

Thank you for everyone contributing with bug reports, feature requests, or pull requests. Feedback and any kind of contribution are always appreciated. Visit syslog-ng’s GitHub page or join Axoflow’s Discord server to reach out to us, and or subscribe to the Axoflow newsletter to receive updates about syslog-ng and our observability and logging-related products.