This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Quick-start guide

This chapter provides a very brief introduction into configuring the AxoSyslog application. For details on the format of the configuration file and how to configure sources, destinations, and other features, refer to the subsequent chapters.

1 - Configuring AxoSyslog on client hosts

Purpose:

To configure AxoSyslog on a client host, complete the following steps.

Steps:

  1. Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.

  2. Configure the local sources to collect the log messages of the host. Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.

    To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.

    • Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
    • Containers: When running AxoSyslog in a container, typically you map an external file to the /etc/syslog-ng/syslog-ng.conf file within the container. Check the mapped volumes of your container, and edit the external file.
    • Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog collector Helm chart.

    Add sources to collect the messages from your log files. File sources look like this:

        source s_myfilesource {
            file("/var/log/myapplication.log" follow-freq(1));
        };
    

    Name every source uniquely. For details on configuring file sources, see file: Collect messages from text files.

  3. Create a network destination that points directly to the AxoSyslog server, or to a local relay. The network destination greatly depends on the protocol that your log server or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:

        destination d_network { network("10.1.2.3" transport("udp")); };
    

    However, if possible, use the much more reliable IETF-syslog protocol over TCP transport:

        destination d_network {
            syslog("10.1.2.3" transport("tcp"));
        };
    
  4. Create a log statement connecting the local sources to the AxoSyslog server or relay. For example:

        log {
            source(s_local); destination(d_network);
        };
    
  5. If the logs will also be stored locally on the host, create local file destinations.

  6. Create a log statement connecting the local sources to the file destination.

  7. Set filters, macros and other features and options (for example, TLS encryption) as necessary.

Example: The default configuration file of AxoSyslog

The following is the default configuration file of AxoSyslog4.5.0. It collects local log messages and the log messages of AxoSyslog and saves them in the /var/log/messages file.

    @version: 4.5.0
    @include "scl.conf"
    source s_local {
        system(); internal();
    };
    destination d_local {
        file("/var/log/messages");
    };
    log {
        source(s_local); destination(d_local);
    };

Example: A simple configuration for clients

The following is a simple configuration file that collects local log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 4.5.0
    @include "scl.conf"
    source s_local {
        system(); internal();
    };
    destination d_syslog_tcp {
        syslog("192.168.1.1" transport("tcp") port(2010));
    };
    log {
        source(s_local);destination(d_syslog_tcp);
    };

2 - Configuring AxoSyslog on server hosts

Purpose:

To configure AxoSyslog on a server host, complete the following steps.

Steps:

  1. Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.

  2. Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.

  3. To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.

    • Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
    • Containers: When running AxoSyslog in a container, typically you map an external file to the /etc/syslog-ng/syslog-ng.conf file within the container. Check the mapped volumes of your container, and edit the external file.
    • Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog collector Helm chart.

    Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:

        source s_network {
            syslog(ip(10.1.2.3) transport("udp"));
        };
    

    However, if possible, use the much more reliable TCP transport:

        source s_network {
            syslog(ip(10.1.2.3) transport("tcp"));
        };
    

    For other options, see syslog: Collect messages using the IETF-syslog protocol and tcp, tcp6, udp, udp6: OBSOLETE - Collect messages from remote hosts using the BSD syslog protocol.

  4. Create local destinations that will store the log messages, for example, file- or program destinations. The default configuration of AxoSyslog places the collected messages into the /var/log/messages file:

        destination d_local {
            file("/var/log/messages");
        };
    

    If you want to create separate logfiles for every client host, use the ${HOST} macro when specifying the filename, for example:

        destination d_local {
            file("/var/log/messages_${HOST}");
        };
    

    For details on further macros and how to use them, see template and rewrite: Format, modify, and manipulate log messages.

  5. Create a log statement connecting the sources to the local destinations.

        log {
            source(s_local); source(s_network); destination(d_local);
        };
    
  6. Set filters, options (for example, TLS encryption) and other advanced features as necessary.

    Example: A simple configuration for servers

    The following is a simple configuration file for AxoSyslog that collects incoming log messages and stores them in a text file.

        @version: 4.5.0
        @include "scl.conf"
        options {
            time-reap(30);
            mark-freq(10);
            keep-hostname(yes);
        };
        source s_local {
            system(); internal();
        };
        source s_network {
            syslog(transport(tcp));
        };
        destination d_logs {
            file(
                "/var/log/syslog-ng/logs.txt"
                owner("root")
                group("root")
                perm(0777)
                );
            };
        log {
            source(s_local); source(s_network); destination(d_logs);
        };
    

3 - Configuring AxoSyslog relays

This section describes how to configure AxoSyslog as a relay.

3.1 - Configuring AxoSyslog on relay hosts

To configure AxoSyslog on a relay host, complete the following steps:

  1. Install the AxoSyslog application on the host. For details on installing AxoSyslog on specific operating systems, see Install AxoSyslog.

  2. Configure the network sources that collect the log messages sent by the clients.

  3. Create a network destination that points to the AxoSyslog server.

  4. Create a log statement connecting the network sources to the AxoSyslog server.

  5. Configure the local sources that collect the log messages of the relay host.

  6. Create a log statement connecting the local sources to the AxoSyslog server.

  7. Enable the keep-hostname() and disable the chain-hostnames() options. (For details on how these options work, see Global options.)

  8. Set filters and options (for example, TLS encryption) as necessary.

Example: A simple configuration for relays

The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 4.5.0
    @include "scl.conf"
    options {
        time-reap(30);
        mark-freq(10);
        keep-hostname(yes);
        chain-hostnames(no);
    };
    source s_local {
        system(); internal();
    };
    source s_network {
        syslog(transport(tcp));
    };
    destination d_syslog_tcp {
        syslog("192.168.1.5" transport("tcp") port(2010));
    };
    log {
        source(s_local); source(s_network);
        destination(d_syslog_tcp);
    };

3.2 - How relaying log messages works

Depending on your exact needs about relaying log messages, there are many scenarios and AxoSyslog options that influence how the log message will look like on the logserver. Some of the most common cases are summarized in the following example:

Consider the following example: client-host > relay > server, where the IP address of client-host is 192.168.1.2. The client-host device sends a syslog message to relay. Depending on the settings of relay, the following can happen.

  • By default, the keep-hostname() option is disabled, so relay writes the IP address of the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discarding any IP address or hostname that was originally in the message.

  • If the keep-hostname() option is enabled on relay, but name resolution is disabled (the use-dns() option is set to no), relay uses the HOST field of the message as-is, which is probably 192.168.1.2.

  • To resolve the 192.168.1.2 IP address to a hostname on relay using a DNS server, use the keep-hostname(no) and use-dns(yes) options. If the DNS server is properly configured and reverse DNS lookup is available for the 192.168.1.2 address, AxoSyslog will rewrite the HOST field of the log message to client-host.

  • The above points apply to the AxoSyslog server (server) as well, so if relay is configured properly, use the keep-hostname(yes) option on server to retain the proper HOST field. Setting keep-hostname(no) on server would result in AxoSyslog rewriting the HOST field to the address of the host that sent the message to server, which is relay in this case.

  • If you cannot or do not want to resolve the 192.168.1.2 IP address on relay, but want to store your log messages on server using the IP address of the original host (that is, client-host), you can enable the spoof-source() option on relay. However, spoof-source() works only under the following conditions:

    • The syslog-ng binary has been compiled with the --enable-spoof-source option.
    • The log messages are sent using the highly unreliable UDP transport protocol. (Extremely unrecommended.)

4 - Sending Kubernetes logs to OpenSearch

The following tutorial shows you how to install AxoSyslog on Kubernetes, deploy OpenSearch to the same cluster, and send Kubernetes logs to OpenSearch.

Prerequisites

You need:

  • A Kubernetes cluster. We used minikube with docker driver and Helm. We used a Ubuntu 22.04 (amd64) machine, but it should work on any system that can run minikube (2 CPUs, 2GB of free memory, 20GB of free disk space).

    The OpenSearch service needs a large mmap count setting, so set it to at least 262144, for example:

    sysctl -w vm.max_map_count=262144
    
  • Helm and kubectl installed.

Generate logs

If you don’t already have an application that generates logs deployed to the Kubernetes cluster, install kube-logging/log-generator to generate sample logs. Complete the following steps.

  1. Add the kube-logging repository to Helm.

    helm repo add kube-logging https://kube-logging.github.io/helm-charts
    

    Expected output:

    "kube-logging" has been added to your repositories
    
  2. Update your repositories.

    helm repo update
    

    The output should look like:

    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "kube-logging" chart repository
    Update Complete. ⎈Happy Helming!⎈
    
  3. Install kube-logging/log-generator.

    helm install --generate-name --wait kube-logging/log-generator
    

    The output should look like:

    NAME: log-generator-1684694629
    LAST DEPLOYED: Sun May 21 20:43:49 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    
  4. Check that the log-generator is running:

    kubectl get pods
    

    The output should look like:

    NAME                                        READY   STATUS        RESTARTS       AGE
    log-generator-1681984863-5946c559b9-ftrrn   1/1     Running       0              8s
    

Set up OpenSearch

  1. Install an OpenSearch cluster with Helm:

    helm repo add opensearch https://opensearch-project.github.io/helm-charts/
    

    Expected output:

    "opensearch" has been added to your repositories
    
  2. Update your repositories.

    helm repo update
    

    The output should look like:

    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "opensearch" chart repository
    Update Complete. ⎈Happy Helming!⎈
    
  3. Install OpenSearch. This step can take a few minutes.

    helm install --generate-name --wait opensearch/opensearch
    
  4. Install the OpenSearch dashboards.

    helm install --generate-name --wait opensearch/opensearch-dashboards
    

    The output should look like:

    NAME: opensearch-dashboards-1684695728
    LAST DEPLOYED: Sun May 21 21:02:09 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    1. Get the application URL by running these commands:
      export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch-dashboards-1684695728" -o jsonpath="{.items[0].metadata.name}")
      export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
      echo "Visit http://127.0.0.1:8080 to use your application"
      kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
    
  5. Now you should have 5 pods. Check that they exist:

    kubectl get pods
    

    The output should look like:

    NAME                                                READY   STATUS    RESTARTS   AGE
    log-generator-1681984863-5946c559b9-ftrrn           1/1     Running   0          3m39s
    opensearch-cluster-master-0                         1/1     Running   0          81s
    opensearch-cluster-master-1                         1/1     Running   0          81s
    opensearch-cluster-master-2                         1/1     Running   0          81s
    opensearch-dashboards-1681999620-59f64f98f7-bjwwh   1/1     Running   0          44s
    
  6. Forward the 5601 port of the OpenSearch Dashboards service (replace the name of the pod with your pod).

    kubectl port-forward <name-of-your-opensearch-dashboards-pod> 8080:5601
    

    The output should look like:

    Forwarding from 127.0.0.1:8080 -> 5601
    Forwarding from [::1]:8080 -> 5601
    
  7. Log in to the dashboard at http://localhost:8080 with admin/admin. You will soon create an Index Pattern here, but first you have to send some logs from syslog-ng.

Set up axosyslog-collector

  1. Add the AxoSyslog Helm repository:

    helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
    helm repo update
    
  2. Create a YAML file (called axoflow-demo.yaml in the examples) to configure the collector.

    config:
      sources:
        kubernetes:
          # Collect kubernetes logs
          enabled: true
      destinations:
        # Send logs to OpenSearch
        opensearch:
          - address: "opensearch-cluster-master"
            index: "test-axoflow-index"
            user: "admin"
            password: "admin"
            tls:
              # Do not validate the server's TLS certificate.
              peerVerify: false
            # Send the syslog fields + the metadata from .k8s.* in JSON format
            template: "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))"
    
  3. Check how the syslog-ng.conf file looks with your custom values:

    helm template -f axoflow-demo.yaml -s templates/config.yaml axosyslog/axosyslog-collector
    

    The output should look like:

    # Source: axosyslog-collector/templates/config.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      labels:
        helm.sh/chart: axosyslog-collector-0.3.0
        app.kubernetes.io/name: axosyslog-collector
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/version: "4.2.0"
        app.kubernetes.io/managed-by: Helm
      name: release-name-axosyslog-collector
    data:
      syslog-ng.conf: |
        @version: current
        @include "scl.conf"
    
        options {
          stats(
            level(1)
          );
        };
    
        log {
          source { kubernetes(); };
          destination {
            elasticsearch-http(
              url("https://opensearch-cluster-master:9200/_bulk")
              index("test-axoflow-index")
              type("")
              template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))")
              user("admin")
              password("admin")
              tls(
                peer-verify(no)
              )
            );
          };
        };    
    
  4. Install the axosyslog-collector chart:

    helm install --generate-name --wait -f axoflow-demo.yaml axosyslog/axosyslog-collector
    

    The output should look like:

    NAME: axosyslog-collector-1682002179
    LAST DEPLOYED: Thu Apr 20 16:49:39 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    1. Watch the axosyslog-collector-1682002179 container start.
    
  5. Check your pods:

    kubectl get pods --namespace=default -l app=axosyslog-collector-1682002179 -w
    kubectl get pods
    

    The output should look like:

    NAME                                                READY   STATUS    RESTARTS   AGE
    log-generator-1681984863-5946c559b9-ftrrn           1/1     Running   0          13m
    opensearch-cluster-master-0                         1/1     Running   0          11m
    opensearch-cluster-master-1                         1/1     Running   0          11m
    opensearch-cluster-master-2                         1/1     Running   0          11m
    opensearch-dashboards-1681999620-59f64f98f7-bjwwh   1/1     Running   0          10m
    axosyslog-collector-1682002179-pjlkn                1/1     Running   0          6s
    

Check the logs in OpenSearch

  1. Open OpenSearch dashboard at http://localhost:8080/app/management/opensearch-dashboards/.

  2. Create an Index Pattern called test-axoflow-index: http://localhost:8080/app/management/opensearch-dashboards/indexPatterns. At Step 2, set the Time field to @timestamp.

    OpenSearch create index pattern for syslog messages screenshot

  3. Now you can see your logs on the Discover view at http://localhost:8080/app/discover. Opening the detailed view for a log entry shows you the fields sent to OpenSearch.

    OpenSearch Kubernetes log messages collected with the AxoSyslog syslog-ng distribution Kubernetes metadata collected with the AxoSyslog syslog-ng distribution

5 - Managing and checking the syslog-ng service on Linux

This section describes how to start, stop and check the status of AxoSyslog service on Linux.

Starting AxoSyslog

To start AxoSyslog, execute the following command as root. For example:

systemctl start syslog-ng

If the service starts successfully, no output will be displayed.

The following message indicates that AxoSyslog can not start (see Checking AxoSyslog status):

Job for syslog-ng.service failed because the control process exited with error code. See `systemctl status syslog-ng.service` and `journalctl -xe` for details.

Stopping AxoSyslog

To stop AxoSyslog

  1. Execute the following command as root.

    systemctl stop syslog-ng

  2. Check the status of AxoSyslog service (see Checking AxoSyslog status).

Restarting AxoSyslog

To restart AxoSyslog, execute the following command as root.

systemctl restart syslog-ng

Reloading configuration file without restarting AxoSyslog

To reload the configuration file without restarting AxoSyslog, execute the following command as root.

systemctl reload syslog-ng

Checking AxoSyslog status

To check the following status-related components, observe the suggestions below.

Checking the status of AxoSyslog service

To check the status of AxoSyslog service

  1. Execute the following command as root.

    systemctl --no-pager status syslog-ng

  2. Check the Active: field, which shows the status of AxoSyslog service. The following statuses are possible:

    • active (running) - syslog-ng service is up and running

          syslog-ng.service - System Logger Daemon
          Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
          Active: active (running) since Tue 2019-06-25 08:58:09 CEST; 5s ago
          Main PID: 6575 (syslog-ng)
          Tasks: 3
          Memory: 13.3M
          CPU: 268ms
          CGroup: /system.slice/syslog-ng.service
          6575 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
      
    • inactive (dead) - syslog-ng service is stopped

          syslog-ng.service - System Logger Daemon
          Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
          Active: inactive (dead) since Tue 2019-06-25 09:14:16 CEST; 2min 18s ago
          Process: 6575 ExecStart=/opt/syslog-ng/sbin/syslog-ng -F --no-caps --enable-core $SYSLOGNG_OPTIONS (code=exited, status=0/SUCCESS)
          Main PID: 6575 (code=exited, status=0/SUCCESS)
          Status: "Shutting down... Tue Jun 25 09:14:16 2019"
          Jun 25 09:14:31 as-syslog-srv systemd: Stopped System Logger Daemon.
      

Checking the process of AxoSyslog

To check the process of AxoSyslog, execute one of the following commands.

  • ps u <pid of syslog-ng>

    Expected output example:

    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
    
  • ps axu | grep syslog-ng | grep -v grep

    Expected output example:

    syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
    

Checking the internal logs of AxoSyslog

The internal logs of AxoSyslog contains informal, warning and error messages.

By default, AxoSyslog log messages (generated on the internal() source) are written to /var/log/messages.

Check the internal logs of AxoSyslog for any issue.

Message processing

The AxoSyslog application collects statistics about the number of processed messages on the different sources and destinations.

Central statistics

To check the central statistics, execute the following command to see the number of received and queued (sent) messages by AxoSyslog.

watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center"

The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is processing the messages. Output example:

    Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center       Tue Jun 25 10:33:25 2019
    center;;queued;a;processed;112
    center;;received;a;processed;28

Source statistics

To check the source statistics, execute the following command to see the number of received messages on the configured sources.

watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source"

The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is receiving messages on the sources. Output example:

    Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source      Tue Jun 25 10:40:50 2019
    source;s_null;;a;processed;0
    source;s_net;;a;processed;0
    source;s_local;;a;processed;90

Destination statistics

To check the source statistics, execute the following command to see the number of received messages on the configured sources.

watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source"

The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is receiving messages on the sources. Output example:

    Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^destination      Tue Jun 25 10:41:02 2019
    destination;d_logserver2;;a;processed;90
    destination;d_messages;;a;processed;180
    destination;d_logserver;;a;processed;90
    destination;d_null;;a;processed;0