syslog-ng

Onboarding allows you to collect metrics about the host, display the host on the Topology page, and to tap into the log flow.

Onboarding requires you to modify the host and the configuration of the logging agent (syslog-ng) running on the host.

Level 1: Install Axolet on the host where syslog-ng is running. Axolet collects metrics from the host and sends them to the AxoConsole, so you can check host-level metrics on the Metrics & Health page of the host, and displays the host on the Topology page.
Level 2: Instrument the configuration of the logging agent to provide detailed metrics about the traffic flow. This allows you to display data about the host on the Analytics page.
Level 3: Instrument the configuration of the logging agent to allow you to access the logs of the logging agent and to tap into the log flow from the AxoConsole. The exact steps for this integration depend on the configuration of your logging agent. We provide basic instrumentation instructions for getting started in this guide, but we strongly recommend to contact us so our professional services can help you with a production integration.

To onboard an existing syslog-ng instance into Axoflow, complete the following steps.

Install Axolet on the host, then approve its registration on the Provisioning page of the AxoConsole.
The syslog-ng host will now be visible on the Topology page of the AxoConsole as a source.
If you've already added the AxoRouter instance or other destination where this newly-registered host is sending data to the AxoConsole, add a path to connect the host to the AxoRouter or the destination.
1. Select Topology > Add Item > Path.
2. Select the target router or aggregator this source is sending its data to in the Target host field, for example, axorouter.
3. Select the Target connector. The connector determines how the destination receives the data (for example, using which protocol or port).
4. Select Add. The new path appears on the Topology page.
Access the syslog-ng host and edit the configuration of syslog-ng. Set the statistics-related global options like this (if the options block already exists, add these lines to the bottom of the block):
```
options {
    stats-level(2);
    stats-freq(0); # Inhibit statistics output to stdout
};
```

(Optional) To get detailed metrics and analytics about the traffic that flows through the host, instrument your syslog-ng configuration as follows:

Note You can use Axolet with an un-instrumented syslog-ng configuration file, but that limits available metrics to host statistics (for example, disk, memory, queue information). You won’t be able to access data about the actual traffic flowing through the host. To collect traffic-related metrics, instrument configuration with metrics-probe() stanzas. The example below shows how to instrument the configuration to highlight common macros such as $HOST and $PROTOCOL. If you want to customize the collected metrics or need help with the instrumentation, contact us.

Download the following configuration snippet to the syslog-ng host, for example, as /etc/syslog-ng/conf.d/axoflow-instrumentation.conf for syslog-ngor /opt/syslog-ng/etc/axoflow-instrumentation.conf for syslog-ng PE.

# Metrics probe parser definition for a specified destination
    # This will yield a reasonable number of metrics for common
    # syslog-ng macros
    block parser metrics-output(destination(undefined)) {
    
    # This will probe for the number of *events*.
    # key name is fixed; labels are user-specified
      metrics-probe(
        key("classified_output_events_total")
        labels(
          "host" => "${HOST}"
          "service" => "${PROGRAM}"
          "source" => "${SOURCE}"
          "dest_port" => "$DESTPORT"
          "dest_ip" => "$DESTIP"
          "src_ip" => "$SOURCEIP"
          "protocol" => "$(if ($PROTO == 6) TCP $(if ($PROTO == 17) UDP $PROTO))"
          "ip_protocol" => "$IP_PROTO"
          "transport" => "$TRANSPORT"
          "destination_name" => "`destination`"
          "issue" => "$(or $(tags-head message.parse_error syslog.unexpected_framing syslog.missing_pri syslog.rfc3164_missing_header syslog.invalid_hostname message.utf8_sanitized) 'none')"
        )
      );
    
    # This will probe for the number of *bytes*.
    # key name is fixed; labels are user-specified
      metrics-probe(
        key("classified_output_event_bytes_total")
        increment("${RAWMSG_SIZE}")
        labels(
            "host" => "${HOST}"
            "service" => "${PROGRAM}"
            "source" => "${SOURCE}"
            "dest_port" => "$DESTPORT"
            "dest_ip" => "$DESTIP"
            "src_ip" => "$SOURCEIP"
            "protocol" => "$(if ($PROTO == 6) TCP $(if ($PROTO == 17) UDP $PROTO))"
            "ip_protocol" => "$IP_PROTO"
            "transport" => "$TRANSPORT"
            "destination_name" => "`destination`"
            "issue" => "$(or $(tags-head message.parse_error syslog.unexpected_framing syslog.missing_pri syslog.rfc3164_missing_header syslog.invalid_hostname message.utf8_sanitized) 'none')"
        )
      );
    };
    
    # For syslog-ng OSE this should be /var/lib/syslog-ng/axotap.sock
    @define AXOTAP_SOCK "/opt/syslog-ng/var/axotap.sock"
    
    destination "d_axotap" {
      channel {
        filter {
          rate-limit(key("$SOURCE:$PROGRAM:$HOST:$SOURCEIP:$DESTIP") rate(1));
        };
    
        rewrite {
          set("$(substr ${MESSAGE} 0 32768)" value(MESSAGE));
          set("$(substr ${RAWMSG} 0 32768)" value(RAWMSG));
          set("$(format-json debug.msg=$(format-json --scope all-nv-pairs) log.body=${MESSAGE})" value(".tap.event"));
          set('{"note":"message object too long for tapping"}' value(".tap.event") condition("$(length ${.tap.event})" > 131072));
          set("$(format-json tap=$(format-json tapId='' eventId=${UNIQID} labels=$(format-json) hostCandidate=$(format-json) facility=${FACILITY} format=${MSGFORMAT} hostFrom=${HOST_FROM} message=$(echo ${MESSAGE}) rawMessage=${RAWMSG} severity=${SEVERITY} source=${SOURCE} tags=${TAGS} timestamp=${ISODATE} transport=${TRANSPORT}) event=json(${.tap.event}))" value(".tap.formatted"));
        };
    
        destination {
          unix-dgram("`AXOTAP_SOCK`" template("${.tap.formatted}"));
        };
      };
    };

Include it in at the top of your configuration file:

@version: current
@include "axoflow-instrumentation.conf"

Edit every destination statement to include a parser { metrics-output(destination(my-destination)); }; line (making sure to include the channel construct if your destination block does not already contain it):

destination my-destination {
    channel {
        parser { metrics-output(destination(my-destination)); };
        destination { file("/dev/null"); };
        };
    };

Reload the configuration of syslog-ng.
```
systemctl reload syslog-ng
```
To enable log tapping for the traffic that flows through the host, add the d_axotap destination in your log paths. This allows the Axolet agent to collect rate-limited log samples from that specific point in the pipeline.

Note There are multiple ways to add tapping to your configuration, which in some cases can introduce unwanted side effects. Contact us to help you create a safe tapping configuration for your use case!
1. Find a log path that you want to tap into and add the tap destination in a safe way, for example, by adding it in a separate inner log path:
```
log {
    source(s_udp514);
    log { destination(d_udp514); };
    log { destination(d_axotap); };
};
```
2. Reload the configuration of syslog-ng.
```
systemctl reload syslog-ng
```
3. Unregister the service so that auto service registration can discover the new tapping point:
4. Test log tapping from the top right menu: