Exploring syslog-ng’s New OpenTelemetry Support: Bridging the Gap for Enhanced Observability

In today’s ever-evolving landscape of logging, tracing, and metrics, OpenTelemetry (OTel) has emerged as the de facto standard for observability. As technologies advance and communication protocols evolve, syslog-ng, a powerful and flexible log management solution, has taken a significant step forward with its integration of OTel support. In this technical blog post, we’ll delve into the realm of syslog-ng’s latest update – version 4.3.1 – and explore how this integration revolutionizes the way logs, traces, and metrics are managed and transported.

Embracing the OTel Ecosystem: Necessity and Opportunity

Syslog-ng has always embraced a multitude of protocols, so its developer community recognized the need for aligning with the OTel ecosystem. As the industry embraces OpenTelemetry as a new standard, syslog-ng has taken a significant step by enabling effortless OTLP/gRPC communication between various observability entities, bringing enhanced benefits to open-source users. This integration opens up new avenues for interaction and data exchange, ensuring that syslog-ng remains at the forefront of modern log management. It also allows OTLP/gRPC communication between syslog-ng instances (more on that in an upcoming blog post).

Parsing, Formatting, and Versatility: Unlocking New Possibilities

The improvements from OTel integration go beyond basic message forwarding, extending to collaborating seamlessly with every processing element and connector within syslog-ng. Syslog-ng offers the ability to parse and format OTel messages, translating them into syslog-ng’s signature name-value pairs. This transformation brings an unparalleled level of versatility.

Sysadmins, developers, and DevOps professionals can craft OTel messages to suit their needs, or even send any syslog-ng message via the OTel protocol. The power to shape data according to specific requirements further solidifies syslog-ng’s role as a flexible bridge between observability systems.

Exploring Practical Application: Demonstrating syslog-ng’s OpenTelemetry Integration

In the following sections of this blog post, we’ll immerse ourselves in the details of syslog-ng’s integration with OpenTelemetry (OTel). We’ll take a closer look at key aspects such as message parsing, offering insights into the capabilities this integration brings. Moreover, we’ll provide hands-on experience by delving into real-world scenarios. We will use AxoSyslog, the cloud-native distribution of syslog-ng by Axoflow.

Imagine this: we’ve harnessed the potential of syslog-ng’s OTel integration in a tangible way. Through modifications to the OpenTelemetry Demo project, we’ve introduced syslog-ng as a central piece in the data flow. Picture two OpenTelemetry Collectors, connected by a syslog-ng instance acting as a bridge. In this forward mode setup, the syslog-ng instance not only seamlessly forwards the data it receives from one Collector to the other, but it also goes a step further. It parses the incoming messages and formats them into JSON and writes them to the standard output. This demonstration, tailored to showcase syslog-ng’s OTel integration, empowers both seasoned syslog-ng users and those new to the world of syslog-ng and OpenTelemetry. By the end of this exploration, you’ll be equipped with the insights needed to harness the full potential of this integration for your observability needs.

For further configurability, like setting up authentication, make sure to check out the AxoSyslog  documentation.

 

Prerequisites

Before you begin, you will need: 

  • a Kubernetes cluster (Minikube, kind, or K3s can help you get started easily),
  • Helm, and
  • a recent kubectl version.

Architecture

“OpenTelemetry Demo is composed of microservices written in different programming languages that talk to each other over gRPC and HTTP; and a load generator which uses Locust to fake user traffic.” – https://opentelemetry.io/docs/demo/architecture/

OpenTelemetry demo architecture
In this orchestrated setup, microservices generate traces and metrics. An OpenTelemetry Collector receives this data, forwarding it to our AxoSyslog instance. AxoSyslog parses and outputs the data to the standard output for us to inspect using the kubectl logs command. Additionally, the data is forwarded to another OpenTelemetry Collector, which feeds Prometheus and Jaeger. Grafana charts visualize the metrics and traces.

This is just for demonstration purposes, in a real world scenario you might be using AxoSyslog to collect security logs or to feed your SIEM. AxoSyslog is capable of sending the received OpenTelemetry messages to any destination and can send messages coming from any source to the other OpenTelemetry Collector. You can also use AxoSyslog’s extensive processing capabilities to parse, transform, filter or route messages.

Installing the demo chart

First, we are going to install the AxoSyslog OpenTelemetry Demo umbrella Helm chart, which is based on the OpenTelemetry Demo project.

helm repo add axosyslog-opentelemetry-demo https://axoflow.github.io/opentelemetry-helm-charts
helm repo update
helm install my-axosyslog-opentelemetry-demo axosyslog-opentelemetry-demo/opentelemetry-demo

Syslog-ng is set up to run with the following configuration. You can retrieve it by running a command similar to this one, just replace the name of the pod:

kubectl exec -it my-axosyslog-opentelemetry-demo-axosyslog-collector-b4vfh -- cat /etc/syslog-ng/syslog-ng.conf
@version: current
@include "scl.conf"

options {
  stats(
    level(2)
  );
};

log {
  source {
    opentelemetry(
        port(4317)
    );
  };
  parser {
    opentelemetry();
  };
  destination {
    file(
      "/dev/stdout"
      template("AxoSyslog - OpenTelemetry ${.otel.type} received: $(format-json .otel.* --shift-levels 1 --include-bytes)\n\n")
    );
  };
  destination {
    opentelemetry(
      url("my-axosyslog-opentelemetry-demo-otelcol:4317")
    );
  };
};

Let’s check AxoSyslog’s logs by running:

kubectl logs svc/my-axosyslog-opentelemetry-demo-axosyslog-collector -f

AxoSyslog - OpenTelemetry metric received: {"otel":{"type":"metric","scope":{"version":"","schema_url":"","name":"spanmetricsconnector","dropped_attributes_count":0},"resource":{"schema_url":"","dropped_attributes_count":0,"attributes":{"telemetry":{"sdk":{"version":"1.8.2","name":"opentelemetry","language":"cpp"}},"service":{"namespace":"opentelemetry-demo","name":"currencyservice","instance":{"id":"e26f74fe-a2b7-4d76-8e77-ff54150cb716"}},"k8s":{"pod":{"name":"my-otel-demo-currencyservice-5d6447cb79-ff5g4"},"node":{"name":"minikube"},"namespace":{"name":"default"}}}},"metric":{"unit":"ms","name":"duration","description":"","data":{"type":"histogram","histogram":{"data_points":{"0":{"time_unix_nano":1692186168832170567,"sum":11.068406000000003,"start_time_unix_nano":1692185403830546303,"min":0,"max":0,"flags":0,"explicit_bounds":{"9":800,"8":400,"7":200,"6":100,"5":50,"4":10,"3":8,"2":6,"15":15000,"14":10000,"13":5000,"12":2000,"11":1400,"10":1000,"1":4,"0":2},"exemplars":{"1":{"value":0.027028,"trace_id":"aHsRcJgsEXY4fUGQj3A2Xg==","time_unix_nano":1692186168832170567,"span_id":"yvxhKkFwil8="},"0":{"value":0.033593999999999999,"trace_id":"aHsRcJgsEXY4fUGQj3A2Xg==","time_unix_nano":1692186168832170567,"span_id":"gzHS2OpbYhI="}},"count":271,"bucket_counts":{"9":0,"8":0,"7":0,"6":0,"5":0,"4":0,"3":0,"2":0,"16":0,"15":0,"14":0,"13":0,"12":0,"11":0,"10":0,"1":0,"0":271},"attributes":{"status":{"code":"STATUS_CODE_OK"},"span":{"name":"CurrencyService/Convert","kind":"SPAN_KIND_SERVER"},"service":{"name":"currencyservice"}}}},"aggregation_temporality":2}}}}}

AxoSyslog - OpenTelemetry span received: {"otel":{"type":"span","span":{"trace_state":"","trace_id":"AuXaz1pTEmNWY6MNpeegog==","status":{"message":"","code":0},"start_time_unix_nano":1692186165572296871,"span_id":"K6hVTPJzc+8=","parent_span_id":"","name":"HTTP GET","kind":3,"end_time_unix_nano":1692186165579437573,"dropped_links_count":0,"dropped_events_count":0,"dropped_attributes_count":0,"attributes":{"http":{"url":"http://my-otel-demo-frontend:8080/api/products/66VCHSJNUP","status_code":200,"method":"GET"}}},"scope":{"version":"0.36b0","schema_url":"","name":"opentelemetry.instrumentation.requests","dropped_attributes_count":0},"resource":{"schema_url":"","dropped_attributes_count":0,"attributes":{"telemetry":{"sdk":{"version":"1.15.0","name":"opentelemetry","language":"python"}},"service":{"namespace":"opentelemetry-demo","name":"loadgenerator","instance":{"id":"f8c7ccdc-62ac-42c7-b45e-4ee1dedfd053"}},"k8s":{"pod":{"name":"my-otel-demo-loadgenerator-d5bbf55ff-mzk9t"},"node":{"name":"minikube"},"namespace":{"name":"default"}}}}}}

You can see that the opentelemetry() parser creates name-value pairs starting with the .otel. prefix, which then are used by the $(format-json) template function to generate a JSON message, which is printed to the standard output.

Port-forward the frontendproxy service by running:

kubectl port-forward svc/my-axosyslog-opentelemetry-demo-frontendproxy 8080:8080

Then check the traces on the Jaeger UI: http://localhost:8080/jaeger/ui/search:

Jaeger UI
You can also check the metrics on the Grafana Dashboards at http://localhost:8080/grafana/d/W2gX2zHVk/demo-dashboard:
Grafana dashboards

Summary

In this post we have shown you that the recent syslog-ng release offers support for receiving and sending OpenTelemetry messages, and demonstrated that it can act as an element in your OpenTelemetry-based observability infrastructure by including it in the OpenTelemetry Demo project architecture. Stay tuned for more blog posts that will show you how to use the syslog-ng OpenTelemetry support in various use cases.

Follow Our Progress!

We are excited to be realizing our vision above with a full Axoflow product suite.

Subscribe for Product News

  • Technology oriented content only.
  • Not more than 1-3 posts per month.
  • You can unsubscribe any time.

By signing up you agree to receive promotional messages
according to Axoflow's Terms of Services.