Multi-tenancy, namespace-based routing, new outputs in Logging operator 4.4

We are pleased to announce that the newest Logging operator release (4.4.0) is available with several notable features. The new release is focused on multi-tenancy capabilities and extends the list of syslog-ng outputs with support for some of the most popular providers.

• Multi-tenancy with namespace-based routing
• New syslog-ng outputs
• Timeout-based configuration checks
• Buffer volume metrics
• New Fluentd 1.16 image

Multi-tenancy with namespace-based routing

Logging operator now supports namespace based routing for efficient aggregator-level multi-tenancy. Using the new LoggingRoute resource, an administrator can define a routing rule that allows for sending logs from one centrally managed Fluent Bit agent to multiple different Fluentd or syslog-ng aggregators owned by individual tenants. That way each tenant receives only the logs relevant to them. Previously this wasn’t possible, tenants had to define their own Fluent Bit agents and receive all logs from the cluster to be able to run an aggregator.

In addition to that, operations teams still have the ability to receive all logs from the system, for example, to archive them on a cold storage. The following diagram demonstrates this use case given two user-level and one administrator-level tenants:

For details, see the following links:

• An overview about multi-tenancy.
• More detailed information about the new LoggingRoute resource.

Note that Logging operator also supports nodegroup level isolation for hard multi-tenancy.

New syslog-ng outputs

When using syslog-ng as the log aggregator, you can now use the following new outputs:

• ElasticSearch
• Grafana Loki
• MongoDB
• Redis
• Amazon S3
• Splunk HEC

Other new syslog-ng features

The HTTP output now supports the log-fifo-size, response-action, and timeout fields.
You can now use the metrics-probe() parser of syslog-ng in syslogNGFLow and SyslogNGClusterFlow. For details, see MetricsProbe.

Forwarder logs 

Fluent Bit now automatically excludes aggregator logs to avoid infinitely growing message loops. With this change, you can access Fluentd and syslog-ng logs simply by running:

kubectl logs <name-of-forwarder-pod>

Timeout-based configuration checks 

Timeout-based configuration checks are different from the normal method: they start a Fluentd or syslog-ng instance without the dry-run or syntax-check flags, so output plugins or destination drivers actually try to establish connections and will fail if there are any issues, for example, with the credentials.

Buffer volume metrics updates

The buffer metrics are now available for both the Fluentd and the SyslogNG based aggregators. The sidecar configuration has been rewritten to add a new metric and improve performance by avoiding unnecessary cardinality. The name of the metric has been changed as well, but the original metric was kept in place to avoid breaking existing clients.

See the following two metrics output how the one is different from the old:

Old:

# HELP node_buffer_size_bytes Disk space used [deprecated]
# TYPE node_buffer_size_bytes gauge
node_buffer_size_bytes{entity="/buffers"} 32253

New:

# HELP logging_buffer_files File count
# TYPE logging_buffer_files gauge
logging_buffer_files{entity="/buffers",host="all-to-file-fluentd-0"} 2
# HELP logging_buffer_size_bytes Disk space used
# TYPE logging_buffer_size_bytes gauge
logging_buffer_size_bytes{entity="/buffers",host="all-to-file-fluentd-0"} 32253

Istio support 

For jobs/individual pods that run to completion, Istio sidecar injection needs to be disabled, otherwise the affected pods would live forever with the running sidecar container. Configuration checkers and Fluentd drainer pods can be configured with the label sidecar.istio.io/inject set to false.

spec:
  configCheck:
    labels:
      sidecar.istio.io/inject: false
  fluentd:
    scaling:
      drain: 
        labels:
          sidecar.istio.io/inject: false

Other improvements

• You can now configure the resources of the buffer metrics sidecar.
• You can now rerun failed configuration checks by deleting the failed configcheck pod.
• The Fluentd ElasticSearch output now supports the composable index template format. To use it, set the use_legacy_template option to false.
• The metrics for the syslog-ng forwarder are now exported using axosyslog-metrics-exporter.

Image updates 

For the list of images used in Logging operator, see Images used by Logging operator.

Support for Fluentd image versions v1.14 and v1.15 is now discontinued because they are based on ruby 2.7 that has reached EOL.

The currently supported image is v1.15-ruby3 and build configuration for v1.15-staging is available for staging experimental changes.

Fluentd v1.16 is available for testing

 A new image based on Fluentd v1.16 is already available to experiment with. To use this new image, change your Logging resource like this:

spec:
  fluentd:
    image:
      tag: v1.16

Although this image is still experimental, please give it a try and let us know how it worked!

Summary

As you can see, Logging operator 4.4 brings you several useful features. You can find the Logging operator release on our GitHub page.

To download the latest Docker image:

docker pull ghcr.io/kube-logging/logging-operator:4.4.0

To install the latest release using Helm:

helm install logging-operator oci://ghcr.io/kube-logging/helm-charts/logging-operator --version=4.4.0

Give it a try!