splunk-hec: Receive messages from Splunk HEC clients

Available in AxoSyslog 4.28 and later.

The splunk-hec() source receives messages from clients that send data to the Splunk HTTP Event Collector (HEC), for example, from Splunk Connect for Syslog (SC4S). This way you can point your existing HEC clients at AxoSyslog without reconfiguring them, and process or route the messages before forwarding them to Splunk or to another destination.

To send messages to Splunk, use the splunk-hec-event() destination instead.

Prerequisites

• Include the AxoSyslog configuration library in your configuration file: @include "scl.conf"
• Install the http and json modules: the axosyslog-mod-http and axosyslog-mod-json packages on Debian and derivatives, or the axosyslog-http and axosyslog-json packages on RHEL and derivatives.

Example: Receiving HEC events

Minimal configuration:

@include "scl.conf"

source s_splunk_hec {
  splunk-hec(
    token("a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d")
  );
};

To receive the events over HTTPS, configure TLS:

source s_splunk_hec_tls {
  splunk-hec(
    token("a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d")
    transport(tls)
    tls(
      key-file("/etc/syslog-ng/hec.key")
      cert-file("/etc/syslog-ng/hec.crt")
    )
  );
};

To test the source, you can use curl on the host where AxoSyslog is running:

curl -X POST -H "Authorization: Splunk a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d" \
  --data '{"time": 1757000000.000, "host": "example-host", "event": "hello from HEC"}' \
  http://127.0.0.1:8088/services/collector

AxoSyslog accepts the request and answers it with the response the HEC clients expect:

{"text":"Success","code":0}

How the messages are processed

The splunk-hec() source handles the incoming requests as follows:

1. It authenticates the request: the Authorization header must be Splunk <token>, where <token> is the value of the token() option. AxoSyslog rejects every other request with 401 Unauthorized.

2. It extracts the events from the body of the request. A request can contain a JSON array of events, or several concatenated JSON objects, and each event becomes a separate log message.

3. It parses each event with the json-parser(), and makes the fields available with the prefix set in the prefix() option, for example, ${.splunk.source} and ${.splunk.sourcetype}.

4. It maps the standard HEC fields to AxoSyslog macros:

   HEC field	Macro
   event	${MESSAGE}
   host	${HOST}
   time	The timestamp of the message. AxoSyslog accepts both the <seconds>.<milliseconds> and the <seconds> epoch formats.

   AxoSyslog maps a field only if it is present and not empty, otherwise the macro keeps its original value.

5. If the payload is not valid JSON, AxoSyslog parses it as a syslog message instead. This way the source also accepts the raw endpoint of HEC.

The source accepts gzip and deflate compressed request bodies.

splunk-hec() source options

The splunk-hec() source has the following options.

chain-hostnames()

Description: Enable or disable the chained hostname format. For details, see the chain-hostnames() global option.

check-hostname()

Description:

When receiving messages, AxoSyslog can check whether the hostname contains valid characters.

Valid characters are:

• alphanumeric characters (A-Z, a-z, 0-9)
• the dash (-) and underscore (_) characters
• the dot (.) and the colon (:) characters
• the @ and slash (/)

If the hostname contains invalid characters, AxoSyslog sets the syslog.invalid_hostname tag for the message, and doesn’t parse the ${HOST} field from the message.

Uses the value of the global option if not specified.

check-program()

Description:

Available in version 4.10 and later.

If the check-program flag is enabled, AxoSyslog validates the ${PROGRAM} field for RFC3164-formatted messages. Valid program names meet the following criteria:

• Contain only these characters: [a-zA-Z0-9-_/().]
• Include at least one alphabetical character.

If the program name fails validation, it’s considered to be part of the log message.

default-facility()

Description: This parameter assigns a facility value to the messages received from the source if the message does not specify one. The default is user (kern for kernel sources).

default-severity()

Description: Assigns a default severity to messages received from this source when the message doesn’t specify one. Accepts the standard syslog severity keywords (for example, emerg, alert, crit, err, warning, notice, info, debug). Combined with default-facility() to compute the message priority.

dns-cache()

Description: Enable or disable DNS cache usage.

host-override()

Description: Replaces the ${HOST} part of the message with the parameter string.

idle-timeout()

Available in AxoSyslog 4.9 and later.

If set, AxoSyslog closes the client connection if no data is received for the specified amount of time (in seconds).

internal()

Description: Marks this pipeline element as internal. Elements marked as internal() are treated as an implementation detail, so for example statistics of the given pipe are available only on higher stats level. This option is mainly useful for developers or when writing SCL blocks and integrations.

ip() or localip()

Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.

ip-freebind()

Description: Linux-only. Enables the IP_FREEBIND socket option, allowing AxoSyslog to bind to a non-local IP address. Useful when the address isn’t yet configured on any interface (for example, during failover scenarios) so the source can listen before the address becomes available. Has no effect on platforms that don’t support IP_FREEBIND.

ip-protocol()

Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).

Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).

For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.

source s_network_tcp { network( transport("tcp") ip("::") ip-protocol(6) port(601) ); };

ip-tos()

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()

Description: Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Description: Specifies whether connections to sources should be closed when AxoSyslog is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the AxoSyslog connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep-hostname()

Description: Enable or disable hostname rewriting.

• If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.

• If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

keep-timestamp()

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

listen-backlog()

Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:

   source s_network {
        network(
            ip("192.168.1.1")
            transport("tcp")
            listen-backlog(2048)
            );
    };

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Description: The size of the source window, that is, the maximum number of in-flight messages the source permits before flow control is enforced. This applies only when flags(flow-control) is set. For details, see Managing incoming and outgoing messages with flow-control.

log-msg-size()

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

This option is deprecated. Use program-override instead.

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

max-connections()

Description: The maximum number of simultaneous client connections that the source accepts.

max-request-size()

Description: The maximum size of a single HTTP request, in bytes. AxoSyslog rejects larger requests with 413 Payload Too Large and logs an error.

For compressed requests, this limit applies to the decompressed body as well.

normalize-hostnames()

Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.

persist-name()

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

port()

Description: The port number to bind to. The default is the default port of the Splunk HTTP Event Collector. Setting transport(tls) doesn’t change this default. Make sure to enable this port on the firewall of the AxoSyslog host.

prefix()

Description: The prefix that AxoSyslog inserts before the name of the fields parsed from the event, to avoid collisions with other name-value pairs of the message. For example, with the default prefix, the sourcetype field of the event becomes available as ${.splunk.sourcetype}.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

program-override()

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

read-old-records()

Description: If set to yes, AxoSyslog will start reading the records from the beginning of the file/journal, if the file/journal has not been read yet. If set to no, AxoSyslog will read only the new records. If the source has a state in the persist file, this option will have no effect.

sdata-prefix()

Available in AxoSyslog 4.1 and later.

Description: Adds a specific string before the names of the parsed SDATA fields to store the name-value pairs created from the SDATA fields separately. Note that unless the value of sdata-prefix starts with .SDATA., using this option excludes the parsed fields from the sdata and rfc5424 scopes of the value pairs.

so-broadcast()

Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.

so-keepalive()

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-rcvbuf()

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so-sndbuf()

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

tcp-keepalive-intvl()

Description: Specifies the interval (number of seconds) between subsequent keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

tcp-keepalive-probes()

Description: Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

tcp-keepalive-time()

Description: Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

time-zone()

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

tls()

Description: The TLS settings of the source, for example, the certificate and the private key that AxoSyslog shows to the clients. Setting transport(tls) requires a tls() block. For details on the individual options, see TLS options.

tls(
    key-file("/etc/syslog-ng/server.key")
    cert-file("/etc/syslog-ng/server.crt")
)

The following TLS options are available: ca-dir(), cert-file(), cipher-suite(), crl-dir(), dhparam-file(), ecdh-curve-list(), key-file(), peer-verify(), pkcs12-file(), ssl-options(), trusted-dn(), and trusted-keys().

token()

Description: The HEC token that the clients must send in the Authorization header of their requests. This is the token that you would otherwise configure on your Splunk deployment, so you can reuse the token your clients already have.

This option is mandatory: if you don’t set it, AxoSyslog rejects every request. Since the token is sent in clear text, use transport(tls) to receive the events over HTTPS.

transport()

Description: Sets whether the source accepts plain HTTP (tcp) or HTTPS (tls) connections. Setting transport(tls) requires a tls() block, and changes the default port to 443.

use-dns()

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from a file (for example, from /etc/hosts) set using the dns-cache-hosts() global option. The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

For details on configuring local DNS resolution, see Using name resolution in syslog-ng.

use-fqdn()

Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.

use-syslogng-pid()

Description: When enabled, AxoSyslog overrides the PID from the original message with the PID of the syslog-ng process.