We are excited to announce Telemetry Controller version 0.0.10! This release simplifies installation, enhances metrics reporting, updates the Loki output, and introduces a new Fluentforward destination to forward data directly to Fluentd.

What is the Telemetry Controller?

The Telemetry Controller is a tool that converts telemetry event streams—logs, metrics, and traces—into Kubernetes resources. It provides an opinionated, convenient, and robust multi-tenant API on top of OpenTelemetry. That way you don’t have to bother with the details, you can just describe what telemetry data you need, and where it should be forwarded. The Telemetry Controller provides isolation and access control for telemetry data, similar to what Kubernetes provides for pods, secrets, and other resources. Plus, it supports multi-tenancy out of the box.

The following are the highlights and main changes of Telemetry controller version 0.0.10.

Easy install via Helm chart

The recommended method to install Telemetry Controller is now via the Helm chart, available both in the GitHub repository as source code and as a package in the GHCR. This chart installs all dependencies, providing a one-command way to install and start using Telemetry Controller.

helm upgrade --install --wait --create-namespace --namespace telemetry-controller-system telemetry-controller oci://ghcr.io/kube-logging/helm-charts/telemetry-controller

To learn more about getting started with Telemetry Controller, see this blog post.

Loki output updates

We’ve updated our Loki output and replaced the underlying OpenTelemetry Collector Loki Exporter with the OpenTelemetry Collector OTLP HTTP Exporter to export OpenTelemetry logs as Loki OTLP endpoint expects them. For details on sending telemetry data to Loki, see the Send Kubernetes logs to Loki with Telemetry Controller blog post.

Output API change

The OtelOutput resources of the earlier versions were renamed to Output, to reflect the types of configurable destinations. In addition to the GRPC based OTLP output, you can now use an HTTP-based OTLP output. This replaces the Loki destination. Besides OTLP, you can send data directly to Fluentd using the fluentforward destination. For example:

apiVersion: telemetry.kube-logging.dev/v1alpha1
kind: Output
metadata:
 name: loki
 namespace: collector
spec:
 loki:
   endpoint: http://loki.loki.svc.cluster.local:3100/otlp/
   tls:
     insecure: true

apiVersion: telemetry.kube-logging.dev/v1alpha1
kind: Output
metadata:
 name: fluent
 namespace: default
spec:
 fluentforward:
   endpoint: all-to-file-fluentd.default:24240
   tag: otelcol
   kubernetes_metadata:
     key: kubernetes
     include_pod_labels: true

Traffic metrics

The Axoflow OTEL Collector distribution (utilized by Telemetry Controller as a default) provides two metric endpoints:

• The default OpenTelemetry Collector endpoint on port 8888. In addition to the default metrics of the deployed OpenTelemetry Collector, we expose fine-grained metrics about data quantity (bytes).  The data is grouped by exporters and receivers. Sample output: 

  # HELP otelcol_exporter_queue_capacity Fixed capacity of the retry queue (in batches)
  # TYPE otelcol_exporter_queue_capacity gauge
  otelcol_exporter_queue_capacity{exporter="otlphttp/collector_loki",service_instance_id="416939ff-c67f-438b-98df-e8180ddcc481",service_name="axoflow-otel-collector",service_version="0.104.0"} 1000
  # HELP otelcol_exporter_queue_size Current size of the retry queue (in batches)
  # TYPE otelcol_exporter_queue_size gauge
  otelcol_exporter_queue_size{exporter="otlphttp/collector_loki",service_instance_id="416939ff-c67f-438b-98df-e8180ddcc481",service_name="axoflow-otel-collector",service_version="0.104.0"} 0
  
  # HELP otelcol_exporter_sent_log_records_bytes Bytes of log records succesfully sent to destination.
  # TYPE otelcol_exporter_sent_log_records_bytes counter
  otelcol_exporter_sent_log_records_bytes{exporter="logging/debug",service_instance_id="416939ff-c67f-438b-98df-e8180ddcc481",service_name="axoflow-otel-collector",service_version="0.104.0"} 4.492622e+06
  otelcol_exporter_sent_log_records_bytes{exporter="otlphttp/collector_loki",service_instance_id="416939ff-c67f-438b-98df-e8180ddcc481",service_name="axoflow-otel-collector",service_version="0.104.0"} 4.492622e+06
  
  ...

• Port 9999 exposes the metrics of the Telemetry Controller pipeline. This data is grouped by tenants and outputs. Sample output metrics:

  # HELP telemetry_controller_output_log_count_total The number of logs sent out from each exporter.
  # TYPE telemetry_controller_output_log_count_total counter
  telemetry_controller_output_log_count_total{exporter="otlphttp/collector_loki",subscription="all-logs",tenant="demo-1"} 2385
  telemetry_controller_output_log_count_total{exporter="otlphttp/collector_loki",subscription="all-logs",tenant="demo-2"} 2380

  Sample tenant metrics:

  # HELP telemetry_controller_tenant_log_count_total The number of logs from each tenant pipeline.
  # TYPE telemetry_controller_tenant_log_count_total counter
  telemetry_controller_tenant_log_count_total{tenant="demo-1"} 2385
  telemetry_controller_tenant_log_count_total{tenant="demo-2"} 2380

Tenant-isolation improvements

Telemetry Controller now allows per-tenant log scraping. This ensures that if one tenant floods the logs, it cannot saturate the pipelines or block log scraping for other tenants.

Summary

The Telemetry Controller 0.0.10 release simplifies the telemetry management process, offering key updates such as easier installation via a Helm chart, updated Loki output, and a more flexible Output API that supports Fluentd as well. In addition, enhanced metrics tracking provides deeper insights into data flow, while tenant-isolation improvements bolster reliability in multi-tenant environments. These advancements strengthen Telemetry Controller’s capabilities, making it a robust solution for Kubernetes telemetry management.

To get started with Telemetry Controller, see this blog post.

Telemetry Controller at KubeCon

If you’re attending Observability Day North America (part of KubeCon) in Salt Lake City on November 12, visit our CTO’s Sandor Guba’s deep-dive talk about advanced multi-tenant use cases, and brand new Telemetry Controller features that we’re releasing in October.