Logging operator 4.8 release

We are happy to announce that Logging operator version 4.8.0 has arrived! The following are the highlights and main changes of Logging operator 4.8. For a complete list of changes and bugfixes, see the Logging operator 4.8 releases page.

Minor breaking change

If you're using hostTailer or eventTailer and configured it through the helm chart's  logging.hostTailer or logging.eventTailer option, note that now both components have an enabled flag. Set this flag to true if you used any of these components from the chart. For details, see the pull request.

Routing based on namespace labels

In your Fluentd ClusterFlows, messages can now be routed based on namespace labels. This highly anticipated feature has been in demand since 2020. Historically, there were workarounds to achieve this, notably through the use of a Fluentd plugin to enhance logs with additional metadata. However, this approach presented two significant challenges:

• Firstly, adding metadata necessitated Kubernetes API Server access on the aggregator node, which was undesirable.
• Secondly, delays in the logging pipeline could lead to inconsistencies in the metadata, as namespace labels might change while the log message was in transit. With the latest major version of Fluent Bit, this issue has been resolved. Fluent Bit is now capable of retrieving and attaching namespace metadata to the log record, enabling the use of this information throughout the logging pipeline.

Use case: multi-tenancy

The primary use case for this feature is to support some form of soft multi-tenancy. In this context, soft multi-tenancy allows for the configuration of a tenant, which is a group of namespaces, as a whole rather than individually. This enables routing of tenant logs to different destinations, or to the same destination while preserving and forwarding the tenant identifier.

There are circumstances where this approach is undesirable, such as when there is a need to provide better isolation between tenants' logs earlier in the logging pipeline. To understand when this typically occurs and how it can be addressed, read our blog post about Log Isolation on Shared Kubernetes Infrastructure.

Getting started

Note that this feature requires the ghcr.io/kube-logging/fluentd:v1.16-4.8-full Fluentd image, which is the default if you use Logging Operator 4.8. If you're using a custom Fluentd image, make sure to update it! Another requirement is Fluent Bit version 3, which is also the default in Logging Operator 4.8.First, enable namespace labeling in Fluent Bit using the FluentBitAgent resource:

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: namespace-label-test
spec:
  filterKubernetes:
    namespace_labels: "On"

The same setting is also available in the logging resource for backwards compatibility:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: namespace-label-test
spec:
  fluentbit:
    filterKubernetes:
      namespace_labels: "On"

After that, you can use namespace labels in your selectors in ClusterFlows, for example:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: dev-logs
spec:
  match:
    - select:
        namespace_labels:
          tenant: dev
  globalOutputRefs:
    - example

Go templates in metrics-probe label values

You can now use go templates that resolve to destination information (name, namespace, scope:local/global, and the logging name) in metrics-probe label values to set custom labels for your metrics. For example:

apiVersion: logging.banzaicloud.io/v1beta1
kind: SyslogNGClusterFlow
metadata:
  name: all
spec:
  match: {}
  outputMetrics:
    - key: custom_output
      labels:
        flow: all
        # define your own label for output name
        my-key-for-the-output: "{{ .Destination.Name }}"
        # do not add the output_name label to the metric
        output_name: ""
  globalOutputRefs:
    - http

Retry limit for Fluent Bit connections

In addition to limiting the number of TCP connections that Fluent Bit workers can open toward the aggregator endpoints (introduced in Logging operator 4.7), you can now also set the Retry_Limit. The following configuration allows Fluent Bit to retry indefinitely on a limited number of connections to avoid overloading the aggregator (syslog-ng in this example).

spec:
  controlNamespace: default
  fluentbit:
  network:
    maxWorkerConnections: 2
  syslogng_output:
    Workers: 2
    Retry_Limit: "no_limits"

Deploy extra manifests

In the Logging operator helm chart you can now include extra manifests that will be deploy together with the chart using the extraManifests field, similarly to other popular charts.

Summary

Routing based on namespace labels has been a much-awaited feature throughout the community, we're happy to finally see it working. Many thanks to all the contributors who help us to keep Logging operator running! You can find the Logging operator release on our GitHub page.To download the latest Docker image:

docker pull ghcr.io/kube-logging/logging-operator:4.8.0

To install the latest release using Helm:

helm install logging-operator oci://ghcr.io/kube-logging/helm-charts/logging-operator --version=4.8.0

Give it a try!