Google Pub/Sub and OpenObserve support in syslog-ng version 4.5

Version 4.5 of syslog-ng gives you exciting new features to better integrate your telemetry pipeline to the services used by your organization.

• Packages are available for various platforms.
• Our AxoSyslog project provides cloud-ready container images and Helm charts.

We cover the highlights of the release in this post, for the in-depth details of every change, read the release notes on the GitHub Releases page. You can also find them in the AxoSyslog documentation.

Google Pub/Sub integration

The google-pubsub() destination feeds your data to Google Pub/Sub via its HTTP REST API. Pub/Sub is an asynchronous and scalable messaging service that is used for streaming analytics and data integration pipelines to ingest and distribute data.

syslog-ng sends events to the Pub/Sub service, and Pub/Sub then delivers events to all the services that react to them. To provide the best integration with Pub/Sub, we have added templating support to the url() option of the http() destination. This allows you to dynamically set the topic, or even the project based on the content or the metadata of the collected data.

Example config:

google-pubsub(
  project("syslog-ng-project")
  topic("syslog-ng-topic")
  auth(
    service-account(
      key("/path/to/service-account-key.json")
    )
  )
);

Sending observability data to OpenObserve

Starting with version 4.5.0, the syslog-ng openobserve-log() destination can send your observability data to OpenObserve using its Logs Ingestion – JSON API. This API accepts multiple records in batches in JSON format.

OpenObserve is an Elasticsearch alternative that provides log search, infrastructure monitoring, and APM solutions. It is a full-fledged, open-source observability platform that can ingest logs, metrics, and traces. It’s compatible with several storage solutions, including S3, GCS, Minio, and Azure Blob, and allows you to significantly decrease storage costs compared to other solutions.

Example config:

openobserve-log(
    url("http://openobserve-endpoint")
    stream("default")
    user("root@example.com")
    password("V2tsn88GhdNTKxaS")
);

URL templating in HTTP

The url() option of the http() destination now supports templates. Syslog-ng templates can only be resolved for a single message, as the same template might have different resolutions for different messages. An http batch consists of multiple messages, so it’s not trivial to decide which message should be used for the resolution.

When batching is enabled and multiple workers are configured, it is important to only batch messages that generate identical URLs. To ensure that, you must set the worker-partition-key() option to use a template that contains all the macros used in the url() option, otherwise messages become mixed.

• For security reasons, syslog-ng automatically applies URL-encoding to the contents in the url() option. Note that the following parts of the url cannot be templated:
  • scheme
  • host
  • port
  • user
  • password

PostgreSQL logs and columnless CSVs

The new postgresql-csvlog-parser() parser processes the CSV logs of PostgreSQL. The data from the CSV-formatted messages is extracted into a set of name-value pairs.

Also, you can now parse CSV messages from files without having to specify the name of the columns. If you omit the columns() parameter of the csv-parser(), the values are extracted into matches ($1, $2, $3 …) (#4678).

cloud-auth() plugin

This new plugin implements different cloud-related authentications. Currently it supports the GCP’s Service Account for the http() destination.

Example config:

http(
  cloud-auth(
	gcp(
	  service-account(
		key("/path/to/service-account-key.json")
		audience("https://pubsub.googleapis.com/google.pubsub.v1.Publisher")
	  )
	)
  )
);

Other changes

• LogSource and LogFetcher: These Python classes now cover explicit source-side batching functionalities (e.g. the auto_close_batch attribute and the close_batch() method).

• rate-limit(): Renamed the template() option to key(), which better communicates the intention.

• templates: The template-escape() option now only escapes the top-level template function. Before syslog-ng 4.5.0 if you had embedded template functions, the template-escape(yes) setting escaped the output of each template function, so the parent template function received an already escaped string. This was never the intention of the template-escape() option.

  Although this is a breaking change, we do not expect anyone having a config that is affected. If you have such a configuration, make sure to follow-up this change. If you need help with it, open an issue or discussion on GitHub, or contact us on the Axoflow Discord server. (#4666)

Summary

This release takes syslog-ng another step closer to becoming a real cloud-native observability tool. We hope you’ll give it a try using the cloud-ready images of AxoSyslog, our syslog-ng distribution.

For the complete list of smaller changes and bug fixes, see the release notes. Stay tuned for more exciting features in the upcoming releases!

Thank you for everyone contributing with bug reports, feature requests, or pull requests. Feedback and any kind of contribution are always appreciated. Visit syslog-ng’s GitHub page or join Axoflow’s Discord server to reach out to us, or subscribe to the Axoflow newsletter to receive updates about syslog-ng and our observability and logging-related products.

For an overview on how our platform enhances syslog-ng based log collection with metrics, see the Metrics for syslog-ng based log management infrastructures blog post.