New metrics and performance improvements in syslog-ng 4.7

• Packages are available for various platforms.
• Our AxoSyslog project provides cloud-ready container images and Helm charts.

We cover the highlights of the release in this post, for the in-depth details of every change, read the release notes on the GitHub Releases page. You can also find them in the AxoSyslog documentation.

opentelemetry() and syslog-ng-otlp() source improvements

We have implemented a number of smaller changes to improve the performance of the opentelemetry() and syslog-ng-otlp() sources:

• concurrent-requests() option:The concurrent-requests() option for the opentelemetry(), syslog-ng-otlp() sources configures the maximal number of in-flight gRPC requests per worker. Setting this value to the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater than or equal to the number of clients, but this can increase the memory usage.
• log-fetch-limit() option: This option can be used to fine tune the performance. To minimize locking while moving messages between source and destination side queues, syslog-ng can move messages in batches. The log-fetch-limit() option sets the maximal size of the batch moved by a worker. By default it is equal to log-iw-size() / workers().
• The log-iw-size() is now split between workers.
• Changed the backpressure behavior. When syslog-ng cannot forward the received message because of backpressure, it no longer returns UNAVAILABLE to the gRPC request. Instead, syslog-ng will block until the destination can accept more messages.

Tags to identify parse errors

Syslog-ng now automatically adds tags that help you identify message parsing errors:

• message.utf8_sanitized
• message.parse_error
• syslog.missing_pri
• syslog.missing_timestamp
• syslog.invalid_hostname
• syslog.unexpected_framing
• syslog.rfc3164_missing_header
• syslog.rfc5424_unquoted_sdata_value

gRPC improvements

channel-args() option in gRPC based drivers

You can now set channel arguments defined in the GRPC Core library documentation in our drivers using the channel-args() option, for example:

opentelemetry(
    channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
    )
);

The following drivers support the new option: bigquery() destination, loki() destination, opentelemetry() source and destination, syslog-ng-otlp() source and destination.

gRPC request metrics

The gRPC based destination drivers now have metrics related to gRPC requests. The affected drivers are: bigquery() destination, opentelemetry(), syslog-ng-otlp(), and loki(). Example metrics:

syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49
syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11

For details, see the AxoSyslog documentation.

Collecting *arr logs

You can collect the logs of the Lidarr, Prowlarr, Radarr, Readarr, and Sonarr (often referred to as “*Arr” or “*Arrs”) applications with the following new *arr() sources:

• lidarr()
• prowlarr()
• radarr()
• readarr()
• sonarr()
• whisparr()

Example minimal config:

source s_radarr {
		radarr(
		dir("/path/to/my/radarr/log/dir")
		);
	};

The logging module is stored in the <prefix><module> name-value pair, for example: .radarr.module => ImportListSyncService. You can modify the prefix with the prefix() option.

Collecting Jellyfin logs

The new jellyfin() source, reads Jellyfin log files. Example minimal config:

source s_jellyfin {
		jellyfin(
		base-dir("/path/to/my/jellyfin/root/log/dir")
		filename-pattern("log_*.log")
		);
	};

For more details about Jellyfin logging, see:

• https://jellyfin.org/docs/general/administration/configuration/#main-configuration
• https://jellyfin.org/docs/general/administration/configuration/#log-directory

As the jellyfin() source is based on the wildcard-file() source, all of the wildcard-file() source options are applicable, too.

New metrics

New http() output metrics

A new counter measures the status of HTTP requests. It is activated on stats(level(1)); for the http() based destination drivers. Example metrics:

syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16
syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2
syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1
syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24

New destination metric

The syslogng_output_event_retries_total counter helps to measure how often a destination fails to receive a message. This counter is available for the following destination drivers: amqp(), bigquery(), http() and all http-based drivers, java(), kafka(), loki(), mongodb(), mqtt(), opentelemetry(), python() and all python-based drivers, redis(), riemann(), smtp(), snmp(), sql(), stomp(), syslog-ng-otlp(). Example metrics:

syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5

Other changes

${TRANSPORT} macro improvements

The ${TRANSPORT} macro now has values for locally created logs. The new values are:

• “local+unix-stream”
• “local+unix-dgram”
• “local+file”
• “local+pipe”
• “local+program”
• “local+devkmsg”
• “local+journal”
• “local+afstreams”
• “local+openbsd”

mqtt() source improvements

The new ${MQTT_TOPIC} name-value pair of the mqtt() source is useful when the name of the topic contains MQTT wildcards. For example:

log {
		source { mqtt(topic("#")); };
		destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); };
	};

Multi-tenancy for loki()

The loki() destination now supports multi-tenancy with the new tenant-id() option. For example:

loki(
		url("localhost:9096")
		labels(
			"app" => "$PROGRAM",
			"host" => "$HOST",
		)
	
		tenant-id("testTenant")
	);

s3() destination improvements

• Support authentication from environment. The access-key() and secret-key() options are now optional, which makes it possible to use authentication methods originated from the environment, e.g. AWS_... environment variables or credentials files from the ~/.aws/ directory. For details, see the official documentation.
• Use default AWS URL if url() is not set.

New template function: $(tags-head)

This template function accepts multiple tag names, and returns the first one that is set. Example config:

# resolves to "bar" if "bar" tag is set, but "foo" is not
template("$(tags-head foo bar baz)")

Packaging changes

APT packages for Debian Buster aren’t built anymore. Old packages are still available, but new syslog-ng versions won’t be available on Debian Buster.

Summary

This release takes syslog-ng another step closer to becoming a real cloud-native observability tool. We hope you’ll give it a try. For the complete list of smaller changes and bug fixes, see the release notes. Stay tuned for more exciting features in the upcoming releases!

Thank you for everyone contributing with bug reports, feature requests, or pull requests. Feedback and any kind of contribution are always appreciated. Visit syslog-ng’s GitHub page or join Axoflow’s Discord server to reach out to us, or subscribe to the Axoflow newsletter to receive updates about syslog-ng and our observability and logging-related products.

For an overview on how our platform enhances syslog-ng based log collection with metrics, see the Metrics for syslog-ng based log management infrastructures blog post.