Sending Kubernetes logs to OpenSearch

The following tutorial shows you how to install AxoSyslog on Kubernetes, deploy OpenSearch to the same cluster, and send Kubernetes logs to OpenSearch.

Prerequisites

You need:

  • A Kubernetes cluster. We used minikube with docker driver and Helm. We used a Ubuntu 22.04 (amd64) machine, but it should work on any system that can run minikube (2 CPUs, 2GB of free memory, 20GB of free disk space).

    The OpenSearch service needs a large mmap count setting, so set it to at least 262144, for example:

    sysctl -w vm.max_map_count=262144
    
  • Helm and kubectl installed.

Generate logs

If you don’t already have an application that generates logs deployed to the Kubernetes cluster, install kube-logging/log-generator to generate sample logs. Complete the following steps.

  1. Add the kube-logging repository to Helm.

    helm repo add kube-logging https://kube-logging.github.io/helm-charts
    

    Expected output:

    "kube-logging" has been added to your repositories
    
  2. Update your repositories.

    helm repo update
    

    The output should look like:

    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "kube-logging" chart repository
    Update Complete. ⎈Happy Helming!⎈
    
  3. Install kube-logging/log-generator.

    helm install --generate-name --wait kube-logging/log-generator
    

    The output should look like:

    NAME: log-generator-1684694629
    LAST DEPLOYED: Sun May 21 20:43:49 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    
  4. Check that the log-generator is running:

    kubectl get pods
    

    The output should look like:

    NAME                                        READY   STATUS        RESTARTS       AGE
    log-generator-1681984863-5946c559b9-ftrrn   1/1     Running       0              8s
    

Set up OpenSearch

  1. Install an OpenSearch cluster with Helm:

    helm repo add opensearch https://opensearch-project.github.io/helm-charts/
    

    Expected output:

    "opensearch" has been added to your repositories
    
  2. Update your repositories.

    helm repo update
    

    The output should look like:

    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "opensearch" chart repository
    Update Complete. ⎈Happy Helming!⎈
    
  3. Install OpenSearch. This step can take a few minutes.

    helm install --generate-name --wait opensearch/opensearch
    
  4. Install the OpenSearch dashboards.

    helm install --generate-name --wait opensearch/opensearch-dashboards
    

    The output should look like:

    NAME: opensearch-dashboards-1684695728
    LAST DEPLOYED: Sun May 21 21:02:09 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    1. Get the application URL by running these commands:
      export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch-dashboards-1684695728" -o jsonpath="{.items[0].metadata.name}")
      export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
      echo "Visit http://127.0.0.1:8080 to use your application"
      kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
    
  5. Now you should have 5 pods. Check that they exist:

    kubectl get pods
    

    The output should look like:

    NAME                                                READY   STATUS    RESTARTS   AGE
    log-generator-1681984863-5946c559b9-ftrrn           1/1     Running   0          3m39s
    opensearch-cluster-master-0                         1/1     Running   0          81s
    opensearch-cluster-master-1                         1/1     Running   0          81s
    opensearch-cluster-master-2                         1/1     Running   0          81s
    opensearch-dashboards-1681999620-59f64f98f7-bjwwh   1/1     Running   0          44s
    
  6. Forward the 5601 port of the OpenSearch Dashboards service (replace the name of the pod with your pod).

    kubectl port-forward <name-of-your-opensearch-dashboards-pod> 8080:5601
    

    The output should look like:

    Forwarding from 127.0.0.1:8080 -> 5601
    Forwarding from [::1]:8080 -> 5601
    
  7. Log in to the dashboard at http://localhost:8080 with admin/admin. You will soon create an Index Pattern here, but first you have to send some logs from syslog-ng.

Set up axosyslog-collector

  1. Add the AxoSyslog Helm repository:

    helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
    helm repo update
    
  2. Create a YAML file (called axoflow-demo.yaml in the examples) to configure the collector.

    config:
      sources:
        kubernetes:
          # Collect kubernetes logs
          enabled: true
      destinations:
        # Send logs to OpenSearch
        opensearch:
          - address: "opensearch-cluster-master"
            index: "test-axoflow-index"
            user: "admin"
            password: "admin"
            tls:
              # Do not validate the server's TLS certificate.
              peerVerify: false
            # Send the syslog fields + the metadata from .k8s.* in JSON format
            template: "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))"
    
  3. Check how the syslog-ng.conf file looks with your custom values:

    helm template -f axoflow-demo.yaml -s templates/config.yaml axosyslog/axosyslog-collector
    

    The output should look like:

    # Source: axosyslog-collector/templates/config.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      labels:
        helm.sh/chart: axosyslog-collector-0.3.0
        app.kubernetes.io/name: axosyslog-collector
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/version: "4.2.0"
        app.kubernetes.io/managed-by: Helm
      name: release-name-axosyslog-collector
    data:
      syslog-ng.conf: |
        @version: current
        @include "scl.conf"
    
        options {
          stats(
            level(1)
          );
        };
    
        log {
          source { kubernetes(); };
          destination {
            elasticsearch-http(
              url("https://opensearch-cluster-master:9200/_bulk")
              index("test-axoflow-index")
              type("")
              template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))")
              user("admin")
              password("admin")
              tls(
                peer-verify(no)
              )
            );
          };
        };    
    
  4. Install the axosyslog-collector chart:

    helm install --generate-name --wait -f axoflow-demo.yaml axosyslog/axosyslog-collector
    

    The output should look like:

    NAME: axosyslog-collector-1682002179
    LAST DEPLOYED: Thu Apr 20 16:49:39 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    1. Watch the axosyslog-collector-1682002179 container start.
    
  5. Check your pods:

    kubectl get pods --namespace=default -l app=axosyslog-collector-1682002179 -w
    kubectl get pods
    

    The output should look like:

    NAME                                                READY   STATUS    RESTARTS   AGE
    log-generator-1681984863-5946c559b9-ftrrn           1/1     Running   0          13m
    opensearch-cluster-master-0                         1/1     Running   0          11m
    opensearch-cluster-master-1                         1/1     Running   0          11m
    opensearch-cluster-master-2                         1/1     Running   0          11m
    opensearch-dashboards-1681999620-59f64f98f7-bjwwh   1/1     Running   0          10m
    axosyslog-collector-1682002179-pjlkn                1/1     Running   0          6s
    

Check the logs in OpenSearch

  1. Open OpenSearch dashboard at http://localhost:8080/app/management/opensearch-dashboards/.

  2. Create an Index Pattern called test-axoflow-index: http://localhost:8080/app/management/opensearch-dashboards/indexPatterns. At Step 2, set the Time field to @timestamp.

    OpenSearch create index pattern for syslog messages screenshot

  3. Now you can see your logs on the Discover view at http://localhost:8080/app/discover. Opening the detailed view for a log entry shows you the fields sent to OpenSearch.

    OpenSearch Kubernetes log messages collected with AxoSyslog Kubernetes metadata collected with AxoSyslog

Last modified June 10, 2024: Leftover tagline updates (8047632)