We are happy to announce that Logging operator version 4.6.0 has arrived, just in time for KubeCon2024 Europe! The highlights include hot configuration reload for Fluent Bit, support for enriching your log messages with Kubernetes namespace labels and annotations, and new VMware outputs.
Kubernetes namespace labels and annotations
Logging operator 4.6 supports the new Fluent Bit Kubernetes filter options that will be released in Fluent Bit 3.0. That way you’ll be able to enrich your logs with Kubernetes namespace labels and annotations right at the source of the log messages. Why is this important? Because multi-tenant solutions (like Capsule or Rancher’s Projects) typically use Kubernetes namespace labels as the identifier of tenants.
Until now the only viable option was to replicate namespace labels on pods using a policy engine (like OPA or Kyverno). This approach worked, but installing and running a policy engine just for this scenario could be an overkill.
With this new Fluent Bit 3.0 feature, you can simply do that in the collector, making it possible to have tenant information right in the log records, which not just enables tenant based filtering in the backend but also enables routing logs to separate backend instances for full isolation.
Fluent Bit 3.0 hasn’t been released yet (at the time of this writing), but you can use a developer image to test the feature, using a FluentbitAgent
resource like this:
apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
name: namespace-label-test
spec:
filterKubernetes:
namespace_annotations: "On"
namespace_labels: "On"
image:
repository: ghcr.io/fluent/fluent-bit/unstable
tag: latest
Fluent Bit hot reload
As a Fluent Bit restart can take a long time when there are many files to index, Logging operator now supports hot reload for Fluent Bit to reload its configuration on the fly.
You can enable hot reloads under the Logging’s spec.fluentbit.configHotReload
(legacy method) option, or the new FluentbitAgent’s spec.configHotReload
option:
apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
name: reload-example
spec:
configHotReload: {}
You can configure the resources
and image
options:
apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
name: reload-example
spec:
configHotReload:
resources: ...
image:
repository: ghcr.io/kube-logging/config-reloader
tag: v0.0.5
Many thanks to @aslafy-z for contributing this feature!
VMware Aria Operations output for Fluentd
When using the Fluentd aggregator with the Logging operator, you can now send your logs to VMware Aria Operations for Logs. This output uses the vmwareLogInsight plugin.
Here is a sample output snippet:
spec:
vmwareLogInsight:
scheme: https
ssl_verify: true
host: MY_LOGINSIGHT_HOST
port: 9543
agent_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
log_text_keys:
- log
- msg
- message
http_conn_debug: false
Many thanks to @logikone for contributing this feature!
VMware Log Intelligence output for Fluentd
When using the Fluentd aggregator with the Logging operator, you can now send your logs to VMware Log Intelligence. This output uses the vmware_log_intelligence plugin.
Here is a sample output snippet:
spec:
vmwarelogintelligence:
endpoint_url: https://data.upgrade.symphony-dev.com/le-mans/v1/streams/ingestion-pipeline-stream
verify_ssl: true
http_compress: false
headers:
content_type: "application/json"
authorization:
valueFrom:
secretKeyRef:
name: vmware-log-intelligence-token
key: authorization
structure: simple
buffer:
chunk_limit_records: 300
flush_interval: 3s
retry_max_times: 3
Many thanks to @zrobisho for contributing this feature!
Other changes
- Enabling ServiceMonitor checks if Prometheus is already available.
- You can now use a custom PVC without a template for the statefulset.
- You can now configure PodDisruptionBudget for Fluentd.
- Event tailer metrics are now automatically exposed.
- You can configure timeout-based configuration checks using the
logging.configCheck
object of thelogging-operator
chart. - You can now specify the event tailer image to use in the
logging-operator
chart. - Fluent Bit can now automatically delete irrecoverable chunks.
- The Fluentd statefulset and its components created by the Logging operator now include the whole securityContext object.
- The Elasticsearch output of the syslog-ng aggregator now supports the template option.
- To avoid problems that might occur when a tenant has a faulty output and backpressure kicks in, Logging operator now creates a dedicated tail input for each tenant.
Removed feature
We have removed support for Pod Security Policies (PSPs), which were deprecated in Kubernetes v1.21, and removed from Kubernetes in v1.25. Note that the API was left intact, it just doesn’t do anything.
Summary
As you can see, Logging operator 4.6 brings you interesting new features. We are especially happy to see so many contributions from the community, many thanks to all of you! You can find the Logging operator release on our GitHub page.
To download the latest Docker image:
docker pull ghcr.io/kube-logging/logging-operator:4.6.0
To install the latest release using Helm:
helm install logging-operator oci://ghcr.io/kube-logging/helm-charts/logging-operator --version=4.6.0
Give it a try!
On-deman Webinar
Parsing
sucks!
What can you do
about it?
56 minutes
Balázs SCHEIDLER
Founder syslog-ng™
Mark BONSACK
Co-creator SC4S
Sándor GUBA
Founder Logging Operator
Neil BOYD
Moderator
On-demand Webinar
Parsing
sucks!
What can you do about it?
56 minutes
Follow Our Progress!
We are excited to be realizing our vision above with a full Axoflow product suite.