OpenTelemetry support in AxoSyslog and syslog-ng

Receive or send any kinds of logs, traces, and metrics with syslog-ng using OTLP/gRPC, improved performance for high-volume connections, and more! Our work on AxoSyslog and syslog-ng has reached a new milestone, and we have released version 4.3.1 with exciting new features that help you bring together your cloud-native and on-premise observability supply chains.

• Packages are available for various platforms.
• Our AxoSyslog project provides cloud-ready container images and Helm charts.

Let’s cover the highlights of the new release. For the in-depth details of every change, read the release notes on the GitHub Releases page.

Receiving and sending OpenTelemetry (OTLP) messages

AxoSyslog now provides support for receiving and sending OpenTelemetry messages using the OTLP/gRPC protocol. You can use the opentelemetry() source, parser and destination to receive, parse and send all kinds of OTLP/gRPC messages: logs, metrics and traces. TLS encryption and multiple authentication methods are supported.

Forwarding OTLP messages

Using the opentelemetry() source and the opentelemetry() destination allows you to use AxoSyslog as a high-performance OpenTelemetry forwarder, with the benefits of using disk-buffering to handle network outages and destination downtimes.

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

Send incoming log messages to OpenTelemetry

Since OpenTelemetry can receive log messages, you can send any log message that syslog-ng collects or receives to OpenTelemetry using the opentelemetry() destination. That way, you can connect your syslog messages to more modern, OpenTelemetry-based observability systems. Message fields that have their type set are sent using the appropriate type.

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

Receive OpenTelemetry messages

Syslog-ng 4.3 can receive all kinds of OTLP/gRPC messages using its opentelemetry() source: logs, metrics and traces. If you use the opentelemetry() parser together with the source, the fields of the incoming messages are available as syslog-ng name-value pairs. The parser maps all the fields (with some limitations).

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

Reliable message transport using OTLP

The OTLP support also allows you to transport messages between nodes in a reliable and secure way. OTLP provides TLS encryption, authentication, and message-level acknowledgement when the messages are received. This results in a functionality based on gRPC that can forward structured messages from the initial receiving AxoSyslog component right up to the central log server, through any number of hops. This functionality is practically identical to that of the syslog-ng driver, but provides better performance.

Nonsequential message processing in syslog-ng

syslog-ng has traditionally performed processing of log messages arriving from a single connection sequentially. This allowed us:

• to ensure message ordering, and to
• efficiently use CPU on a per message basis.

This mode of operation performs well if you have relatively many parallel connections, in which case it uses all the available CPU cores.

However, if a small number of connections deliver a large number of messages, this behavior becomes a bottleneck. With the new parallelization feature, syslog-ng can split a stream of incoming messages into a set of partitions, which can be processed by multiple threads in parallel. Depending on how you partition the stream, you might lose the message ordering, but can scale the incoming load to all CPUs in the system, even if the entire load is coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your log path. The following example takes the messages of the tcp() source and processes them with 4 parallel threads, regardless of the number of connections used to deliver the messages to the tcp() source.

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

parallelize() uses round-robin to allocate messages to partitions by default, but you can retain ordering for a subset of messages with the partition-key() option. The partition-key() option specifies a template: messages that expand the template to the same value are mapped to the same partition. For example, you can partition messages based on their sender host:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };

  destination { ... };
};

CrowdStrike Falcon LogScale (Humio) integration

The new logscale() destination feeds LogScale via its Ingest API: simply create an Ingest token, and send structured events to LogScale:

destination d_logscale {
  logscale(
    token("your-token")
  );
};

Other minor features

Among other minor improvements and bugfixes, we have also:

• Improved the MongoDB support with a bulk insert feature. Just enable bulk(yes) in your mongodb() destination.
• Added support for custom quoting in table and index names (because MySQL sometimes needs this for certain identifiers).

Summary

This release, and especially the OpenTelemetry support, takes syslog-ng another step closer to becoming a real cloud-native observability tool. We hope you’ll give it a try using the cloud-ready images of AxoSyslog, our syslog-ng distribution.

For the complete list of smaller changes and bugfixes, see the release notes. Stay tuned for more exciting features in the upcoming releases!

Thank you for everyone contributing with bug reports, feature requests, or pull requests. Feedback and any kind of contribution are always appreciated. Visit syslog-ng’s GitHub page or join Axoflow’s Discord server to reach out to us, and or subscribe to the Axoflow newsletter to receive updates about syslog-ng and our observability and logging-related products.