Minor breaking change

Routing based on namespace labels

Use case: multi-tenancy

The primary use case for this feature is to support some form of soft multi-tenancy. In this context, soft multi-tenancy allows for the configuration of a tenant, which is a group of namespaces, as a whole rather than individually. This enables routing of tenant logs to different destinations, or to the same destination while preserving and forwarding the tenant identifier.

Getting started

Note that this feature requires the ghcr.io/kube-logging/fluentd:v1.16-4.8-full Fluentd image, which is the default if you use Logging Operator 4.8. If you’re using a custom Fluentd image, make sure to update it! Another requirement is Fluent Bit version 3, which is also the default in Logging Operator 4.8.

First, enable namespace labeling in Fluent Bit using the FluentBitAgent resource:

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: namespace-label-test
spec:
  filterKubernetes:
    namespace_labels: "On"

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: namespace-label-test
spec:
  fluentbit:
    filterKubernetes:
      namespace_labels: "On"

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: dev-logs
spec:
  match:
    - select:
        namespace_labels:
          tenant: dev
  globalOutputRefs:
    - example

Go templates in metrics-probe label values

apiVersion: logging.banzaicloud.io/v1beta1
kind: SyslogNGClusterFlow
metadata:
  name: all
spec:
  match: {}
  outputMetrics:
    - key: custom_output
      labels:
        flow: all
        # define your own label for output name
        my-key-for-the-output: "{{ .Destination.Name }}"
        # do not add the output_name label to the metric
        output_name: ""
  globalOutputRefs:
    - http

Retry limit for Fluent Bit connections

In addition to limiting the number of TCP connections that Fluent Bit workers can open toward the aggregator endpoints (introduced in Logging operator 4.7), you can now also set the Retry_Limit. The following configuration allows Fluent Bit to retry indefinitely on a limited number of connections to avoid overloading the aggregator (syslog-ng in this example).

spec:
  controlNamespace: default
  fluentbit:
  network:
    maxWorkerConnections: 2
  syslogng_output:
    Workers: 2
    Retry_Limit: "no_limits"

Summary

Routing based on namespace labels has been a much-awaited feature throughout the community, we’re happy to finally see it working. Many thanks to all the contributors who help us to keep Logging operator running! You can find the Logging operator release on our GitHub page.

To download the latest Docker image:

docker pull ghcr.io/kube-logging/logging-operator:4.8.0

To install the latest release using Helm:

helm install logging-operator oci://ghcr.io/kube-logging/helm-charts/logging-operator --version=4.8.0

Give it a try!