Exploring syslog-ng’s New OpenTelemetry Support: Bridging the Gap for Enhanced Observability
In today’s ever-evolving landscape of logging, tracing, and metrics, OpenTelemetry (OTel) has emerged as the de facto standard for observability. As technologies advance and communication protocols evolve, syslog-ng, a powerful and flexible log management solution, has taken a significant step forward with its integration of OTel support. In this technical blog post, we’ll delve into the realm of syslog-ng’s latest update – version 4.3.1 – and explore how this integration revolutionizes the way logs, traces, and metrics are managed and transported.
Embracing the OTel Ecosystem: Necessity and Opportunity
Syslog-ng has always embraced a multitude of protocols, so its developer community recognized the need for aligning with the OTel ecosystem. As the industry embraces OpenTelemetry as a new standard, syslog-ng has taken a significant step by enabling effortless OTLP/gRPC communication between various observability entities, bringing enhanced benefits to open-source users. This integration opens up new avenues for interaction and data exchange, ensuring that syslog-ng remains at the forefront of modern log management. It also allows OTLP/gRPC communication between syslog-ng instances (more on that in an upcoming blog post).
Parsing, Formatting, and Versatility: Unlocking New Possibilities
The improvements from OTel integration go beyond basic message forwarding, extending to collaborating seamlessly with every processing element and connector within syslog-ng. Syslog-ng offers the ability to parse and format OTel messages, translating them into syslog-ng’s signature name-value pairs. This transformation brings an unparalleled level of versatility.
Sysadmins, developers, and DevOps professionals can craft OTel messages to suit their needs, or even send any syslog-ng message via the OTel protocol. The power to shape data according to specific requirements further solidifies syslog-ng’s role as a flexible bridge between observability systems.
Exploring Practical Application: Demonstrating syslog-ng’s OpenTelemetry Integration
In the following sections of this blog post, we’ll immerse ourselves in the details of syslog-ng’s integration with OpenTelemetry (OTel). We’ll take a closer look at key aspects such as message parsing, offering insights into the capabilities this integration brings. Moreover, we’ll provide hands-on experience by delving into real-world scenarios. We will use AxoSyslog, the scalable security data processor by Axoflow.
Imagine this: we’ve harnessed the potential of syslog-ng’s OTel integration in a tangible way. Through modifications to the OpenTelemetry Demo project, we’ve introduced syslog-ng as a central piece in the data flow. Picture two OpenTelemetry Collectors, connected by a syslog-ng instance acting as a bridge. In this forward mode setup, the syslog-ng instance not only seamlessly forwards the data it receives from one Collector to the other, but it also goes a step further. It parses the incoming messages and formats them into JSON and writes them to the standard output. This demonstration, tailored to showcase syslog-ng’s OTel integration, empowers both seasoned syslog-ng users and those new to the world of syslog-ng and OpenTelemetry. By the end of this exploration, you’ll be equipped with the insights needed to harness the full potential of this integration for your observability needs.
For further configurability, like setting up authentication, make sure to check out the AxoSyslog documentation.
Prerequisites
Before you begin, you will need:
- a Kubernetes cluster (Minikube, kind, or K3s can help you get started easily),
- Helm, and
- a recent kubectl version.
Architecture
“OpenTelemetry Demo is composed of microservices written in different programming languages that talk to each other over gRPC and HTTP; and a load generator which uses Locust to fake user traffic.” – https://opentelemetry.io/docs/demo/architecture/

kubectl logs
command. Additionally, the data is forwarded to another OpenTelemetry Collector, which feeds Prometheus and Jaeger. Grafana charts visualize the metrics and traces.
This is just for demonstration purposes, in a real world scenario you might be using AxoSyslog to collect security logs or to feed your SIEM. AxoSyslog is capable of sending the received OpenTelemetry messages to any destination and can send messages coming from any source to the other OpenTelemetry Collector. You can also use AxoSyslog’s extensive processing capabilities to parse, transform, filter or route messages.

Installing the demo chart
First, we are going to install the AxoSyslog OpenTelemetry Demo umbrella Helm chart, which is based on the OpenTelemetry Demo project.
helm repo add axosyslog-opentelemetry-demo https://axoflow.github.io/opentelemetry-helm-charts
helm repo update
helm install my-axosyslog-opentelemetry-demo axosyslog-opentelemetry-demo/opentelemetry-demo
Syslog-ng is set up to run with the following configuration. You can retrieve it by running a command similar to this one, just replace the name of the pod:
kubectl exec -it my-axosyslog-opentelemetry-demo-axosyslog-collector-b4vfh -- cat /etc/syslog-ng/syslog-ng.conf
@version: current
@include "scl.conf"
options {
stats(
level(2)
);
};
log {
source {
opentelemetry(
port(4317)
);
};
parser {
opentelemetry();
};
destination {
file(
"/dev/stdout"
template("AxoSyslog - OpenTelemetry ${.otel.type} received: $(format-json .otel.* --shift-levels 1 --include-bytes)\n\n")
);
};
destination {
opentelemetry(
url("my-axosyslog-opentelemetry-demo-otelcol:4317")
);
};
};
Let’s check AxoSyslog’s logs by running:
kubectl logs svc/my-axosyslog-opentelemetry-demo-axosyslog-collector -f
AxoSyslog - OpenTelemetry metric received: {"otel":{"type":"metric","scope":{"version":"","schema_url":"","name":"spanmetricsconnector","dropped_attributes_count":0},"resource":{"schema_url":"","dropped_attributes_count":0,"attributes":{"telemetry":{"sdk":{"version":"1.8.2","name":"opentelemetry","language":"cpp"}},"service":{"namespace":"opentelemetry-demo","name":"currencyservice","instance":{"id":"e26f74fe-a2b7-4d76-8e77-ff54150cb716"}},"k8s":{"pod":{"name":"my-otel-demo-currencyservice-5d6447cb79-ff5g4"},"node":{"name":"minikube"},"namespace":{"name":"default"}}}},"metric":{"unit":"ms","name":"duration","description":"","data":{"type":"histogram","histogram":{"data_points":{"0":{"time_unix_nano":1692186168832170567,"sum":11.068406000000003,"start_time_unix_nano":1692185403830546303,"min":0,"max":0,"flags":0,"explicit_bounds":{"9":800,"8":400,"7":200,"6":100,"5":50,"4":10,"3":8,"2":6,"15":15000,"14":10000,"13":5000,"12":2000,"11":1400,"10":1000,"1":4,"0":2},"exemplars":{"1":{"value":0.027028,"trace_id":"aHsRcJgsEXY4fUGQj3A2Xg==","time_unix_nano":1692186168832170567,"span_id":"yvxhKkFwil8="},"0":{"value":0.033593999999999999,"trace_id":"aHsRcJgsEXY4fUGQj3A2Xg==","time_unix_nano":1692186168832170567,"span_id":"gzHS2OpbYhI="}},"count":271,"bucket_counts":{"9":0,"8":0,"7":0,"6":0,"5":0,"4":0,"3":0,"2":0,"16":0,"15":0,"14":0,"13":0,"12":0,"11":0,"10":0,"1":0,"0":271},"attributes":{"status":{"code":"STATUS_CODE_OK"},"span":{"name":"CurrencyService/Convert","kind":"SPAN_KIND_SERVER"},"service":{"name":"currencyservice"}}}},"aggregation_temporality":2}}}}}
AxoSyslog - OpenTelemetry span received: {"otel":{"type":"span","span":{"trace_state":"","trace_id":"AuXaz1pTEmNWY6MNpeegog==","status":{"message":"","code":0},"start_time_unix_nano":1692186165572296871,"span_id":"K6hVTPJzc+8=","parent_span_id":"","name":"HTTP GET","kind":3,"end_time_unix_nano":1692186165579437573,"dropped_links_count":0,"dropped_events_count":0,"dropped_attributes_count":0,"attributes":{"http":{"url":"http://my-otel-demo-frontend:8080/api/products/66VCHSJNUP","status_code":200,"method":"GET"}}},"scope":{"version":"0.36b0","schema_url":"","name":"opentelemetry.instrumentation.requests","dropped_attributes_count":0},"resource":{"schema_url":"","dropped_attributes_count":0,"attributes":{"telemetry":{"sdk":{"version":"1.15.0","name":"opentelemetry","language":"python"}},"service":{"namespace":"opentelemetry-demo","name":"loadgenerator","instance":{"id":"f8c7ccdc-62ac-42c7-b45e-4ee1dedfd053"}},"k8s":{"pod":{"name":"my-otel-demo-loadgenerator-d5bbf55ff-mzk9t"},"node":{"name":"minikube"},"namespace":{"name":"default"}}}}}}
You can see that the opentelemetry()
parser creates name-value pairs starting with the .otel.
prefix, which then are used by the $(format-json) template function to generate a JSON message, which is printed to the standard output.
Port-forward the frontendproxy service by running:
kubectl port-forward svc/my-axosyslog-opentelemetry-demo-frontendproxy 8080:8080
Then check the traces on the Jaeger UI: http://localhost:8080/jaeger/ui/search:


Summary
In this post we have shown you that the recent syslog-ng release offers support for receiving and sending OpenTelemetry messages, and demonstrated that it can act as an element in your OpenTelemetry-based observability infrastructure by including it in the OpenTelemetry Demo project architecture. Stay tuned for more blog posts that will show you how to use the syslog-ng OpenTelemetry support in various use cases.

Follow Our Progress!
We are excited to be realizing our vision above with a full Axoflow product suite.
