Rancher adopted the Logging operator several years ago and integrated it into its UI for managing logging Flows and Outputs conveniently. That was an easy way for users to manage their logging configuration in a multi-tenant fashion (Rancher uses a concept which they call “Projects” for Kubernetes multi-tenancy, see: Projects and Kubernetes Namespaces with Rancher).

Rancher has since moved their focus on building a different solution which is geared towards their multi-cluster observability use cases. Logging operator is still there and available, but it has not been upgraded since version 3.17 and they have no immediate plans to fix that even though we have reached out and assured them of our support. For users who would still want to use the Logging operator and leverage the latest updates, this is a bummer. 

Existing features of Logging operator our users love:

• Multi-tenant configuration.
• Huge number of available destinations.
• Flexible filtering, transformations and enrichment of log messages.
• Managed, fail-safe configuration upgrades.

Here are the notable new features why you would want to use the latest Logging operator:

• syslog-ng is supported as an aggregator in place of Fluentd for better performance and more flexibility. You can read more about it in the original release blog or in the documentation. (The list of supported syslog-ng outputs is growing fast)
• Separate FluentbitAgent CRD for node level hard multi-tenancy scenarios and to support gradual configuration upgrades.

Also there are two important milestones ahead of us:

• Coming with version 4.4: log routing on the agent for aggregator-level hard multi-tenancy. This feature is going to allow the creation of tenants where each tenant receives logs from its own namespaces while it is still in control of its own aggregator, being it Fluentd or syslog-ng. For example, it should play together with Rancher’s multi-tenant Projects nicely.
• OpenTelemetry support: we are investigating how the Logging operator could leverage the OpenTelemetry collector to make log routing on the node level more flexible with a focus on multi-tenant use cases.

This blog post was inspired by Frank “eumel8” Kloeker’s post on his migration experiences and aims to summarize the recommended way of executing such an exercise.

Installing the Logging Operator from scratch

There is a chance that the Logging app is not enabled in Rancher initially. In that case you can simply start from scratch. Please follow the official installation instructions.

Note: Starting with version 4.3.0, the Logging operator Helm chart is distributed through an OCI registry, which does not use traditional Helm chart repositories anymore. Unfortunately, the first class support for OCI registries has not yet arrived to Rancher as of the time of writing this post. (There is a tracking issue as a reference.)

Upgrading an existing Rancher Logging

In case Logging is already enabled in your Rancher setup, you will have to disable the corresponding apps (rancher-logging and rancher-logging-crds) in a way that is most suitable for your use case.

• Deleting both apps is the most dangerous choice as it will remove every related logging resource and bring down any existing logging pipelines.

  Rancher Logging can have extra log tailers (e.g. for host journald logs) running, depending on how you installed the app. If you need those, continue with the next section.

• If you don’t need the resources in the rancher-logging app but have existing logging resources configured separately, you can remove the rancher-logging app and keep the rancher-logging-crd app only.
• In case you are absolutely sure you don’t have anything to lose by removing both apps, simply delete these and install Logging operator from scratch (see previous section).

Let’s look at a safer option to disable Rancher logging.

Disable Rancher’s Logging operator

To upgrade an existing Rancher Logging deployment, you have to:

1. Stop and remove the Helm managed Logging operator deployment.
2. Remove the helm release secret so that the original resources (running log tailers and logging CRDs) that your system might depend on won’t get removed accidentally.

Note: This will only remove the Rancher logging operator specifically. In case you want to look at all the installed resources, you can do that on the UI, where you can even delete some of them manually:

kubectl delete -n cattle-logging-system deploy/rancher-logging

Let’s continue with backing up and deleting the Helm release secrets, so that the UI forget about the apps.

Note: Don’t worry, you can restore the original resources any time by restoring the Helm release secrets and executing an upgrade on the restored apps from the UI.

# backup the original release secrets
kubectl get secret -n cattle-logging-system -l name=rancher-logging,owner=helm -o yaml > rancher-logging-secret.yaml
kubectl get secret -n cattle-logging-system -l name=rancher-logging-crd,owner=helm -o yaml > rancher-logging-crd-secret.yaml

# delete the release secrets to remove the app
# from the Rancher UI without removing the CRDs or other resources
kubectl delete secret -n cattle-logging-system -l name=rancher-logging,owner=helm
kubectl delete secret -n cattle-logging-system -l name=rancher-logging-crd,owner=helm

You are ready to install the stock Logging operator.

Upgrade the CRDs

Within Helm 3 there is still no option to upgrade CRDs automatically, but GitOps tools like ArgoCD can do it for you. In case you have to do that manually here is a command to execute as a cluster administrator:

helm show crds oci://ghcr.io/kube-logging/helm-charts/logging-operator | kubectl apply --server-side --force-conflicts -f-

The server-side flag for the kubectl apply command is important, because without it the resources would be too big to be applied normally.

Install the new Logging Operator

You can now follow the original installation instructions, because the CRDs are already upgraded and the Rancher managed Logging operator is already removed:

https://kube-logging.dev/docs/install/

After you have completed the installation, the Logging operator is installed in its own namespace and all your existing logging resources in the cluster should operate normally. You have the same Logging menu in Rancher and can set up resources like Logging, Flow, and Output, and use the exciting new features of Logging operator.

Happy Logging!