elasticsearch-bulk: Receive messages from Elastic clients

Available in AxoSyslog 4.28 and later.

The elasticsearch-bulk() source implements the Elasticsearch Bulk API. Elastic Agent, Beats, and other clients of the Bulk API can send their events to AxoSyslog by pointing their Elasticsearch output at this source, so you can process and route the events before forwarding them to Elasticsearch or to another destination.

Example: Receiving events from Beats

source s_es {
  elasticsearch-bulk(
    port(9200)
    auth-token("ApiKey <key>")
    version("8.15.0")
  );
};

Configure the Elasticsearch output of your client to point at this address, for example, in a Filebeat configuration:

output.elasticsearch:
  hosts: ["http://<IP-of-AxoSyslog>:9200"]
  api_key: "<key>"

How the messages are processed

The body of a bulk request is a sequence of action lines, each followed by a document line. AxoSyslog handles the actions as follows:

Because bulk documents are JSON and not syslog messages, this source doesn’t parse the documents as syslog. The whole document line becomes the ${MESSAGE} of the log message.

AxoSyslog treats a request as a bulk request if its method is POST or PUT, and its path ends with /_bulk. It answers the requests of the clients as follows:

Every response includes the X-Elastic-Product: Elasticsearch header, which the Elastic clients require.

The source accepts gzip and deflate compressed request bodies, and the max-request-size() limit applies to the decompressed body as well.

elasticsearch-bulk() source options

The elasticsearch-bulk() source has the following options.

auth-token()

Description: Requires the clients to authenticate themselves with a token. When set, AxoSyslog accepts only the requests whose Authorization header exactly matches the value of this option, and rejects every other request with 401 Unauthorized.

Include the authentication scheme in the value, because AxoSyslog compares the whole header:

auth-token("Bearer s3cr3t")

Since the token is sent in clear text, use this option only over HTTPS, that is, together with transport(tls).

chain-hostnames()

Description: Enable or disable the chained hostname format. For details, see the chain-hostnames() global option.

check-hostname()

Description:

When receiving messages, AxoSyslog can check whether the hostname contains valid characters.

Valid characters are:

• alphanumeric characters (A-Z, a-z, 0-9)
• the dash (-) and underscore (_) characters
• the dot (.) and the colon (:) characters
• the @ and slash (/)

If the hostname contains invalid characters, AxoSyslog sets the syslog.invalid_hostname tag for the message, and doesn’t parse the ${HOST} field from the message.

Uses the value of the global option if not specified.

check-program()

Description:

Available in version 4.10 and later.

If the check-program flag is enabled, AxoSyslog validates the ${PROGRAM} field for RFC3164-formatted messages. Valid program names meet the following criteria:

• Contain only these characters: [a-zA-Z0-9-_/().]
• Include at least one alphabetical character.

If the program name fails validation, it’s considered to be part of the log message.

default-facility()

Description: This parameter assigns a facility value to the messages received from the source if the message does not specify one. The default is user (kern for kernel sources).

default-severity()

Description: Assigns a default severity to messages received from this source when the message doesn’t specify one. Accepts the standard syslog severity keywords (for example, emerg, alert, crit, err, warning, notice, info, debug). Combined with default-facility() to compute the message priority.

dns-cache()

Description: Enable or disable DNS cache usage.

host-override()

Description: Replaces the ${HOST} part of the message with the parameter string.

idle-timeout()

Available in AxoSyslog 4.9 and later.

If set, AxoSyslog closes the client connection if no data is received for the specified amount of time (in seconds).

internal()

Description: Marks this pipeline element as internal. Elements marked as internal() are treated as an implementation detail, so for example statistics of the given pipe are available only on higher stats level. This option is mainly useful for developers or when writing SCL blocks and integrations.

ip() or localip()

Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.

ip-freebind()

Description: Linux-only. Enables the IP_FREEBIND socket option, allowing AxoSyslog to bind to a non-local IP address. Useful when the address isn’t yet configured on any interface (for example, during failover scenarios) so the source can listen before the address becomes available. Has no effect on platforms that don’t support IP_FREEBIND.

ip-protocol()

Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).

Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).

For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.

source s_network_tcp { network( transport("tcp") ip("::") ip-protocol(6) port(601) ); };

ip-tos()

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()

Description: Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Description: Specifies whether connections to sources should be closed when AxoSyslog is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the AxoSyslog connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep-hostname()

Description: Enable or disable hostname rewriting.

• If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.

• If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

keep-timestamp()

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

listen-backlog()

Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:

   source s_network {
        network(
            ip("192.168.1.1")
            transport("tcp")
            listen-backlog(2048)
            );
    };

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Description: The size of the source window, that is, the maximum number of in-flight messages the source permits before flow control is enforced. This applies only when flags(flow-control) is set. For details, see Managing incoming and outgoing messages with flow-control.

log-msg-size()

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

This option is deprecated. Use program-override instead.

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

max-connections()

Description: The maximum number of simultaneous client connections that the source accepts.

max-request-size()

Description: The maximum size of a single HTTP request, in bytes. AxoSyslog rejects larger requests with 413 Payload Too Large and logs an error.

For compressed requests, this limit applies to the decompressed body as well.

normalize-hostnames()

Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.

persist-name()

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

port() or localport()

Description: The port number to bind to. Make sure to enable this port on the firewall of the AxoSyslog host.

program-override()

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

read-old-records()

Description: If set to yes, AxoSyslog will start reading the records from the beginning of the file/journal, if the file/journal has not been read yet. If set to no, AxoSyslog will read only the new records. If the source has a state in the persist file, this option will have no effect.

sdata-prefix()

Available in AxoSyslog 4.1 and later.

Description: Adds a specific string before the names of the parsed SDATA fields to store the name-value pairs created from the SDATA fields separately. Note that unless the value of sdata-prefix starts with .SDATA., using this option excludes the parsed fields from the sdata and rfc5424 scopes of the value pairs.

so-broadcast()

Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.

so-keepalive()

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-rcvbuf()

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so-sndbuf()

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

tcp-keepalive-intvl()

Description: Specifies the interval (number of seconds) between subsequent keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

tcp-keepalive-probes()

Description: Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

tcp-keepalive-time()

Description: Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

time-zone()

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

tls()

Description: The TLS settings of the source, for example, the certificate and the private key that AxoSyslog shows to the clients. Setting transport(tls) requires a tls() block. For details on the individual options, see TLS options.

tls(
    key-file("/etc/syslog-ng/server.key")
    cert-file("/etc/syslog-ng/server.crt")
)

The following TLS options are available: ca-dir(), cert-file(), cipher-suite(), crl-dir(), dhparam-file(), ecdh-curve-list(), key-file(), peer-verify(), pkcs12-file(), ssl-options(), trusted-dn(), and trusted-keys().

transport()

Description: Sets whether the source accepts plain HTTP (tcp) or HTTPS (tls) connections. Setting transport(tls) requires a tls() block, and changes the default port to 443.

use-dns()

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from a file (for example, from /etc/hosts) set using the dns-cache-hosts() global option. The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

For details on configuring local DNS resolution, see Using name resolution in syslog-ng.

use-fqdn()

Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.

use-syslogng-pid()

Description: When enabled, AxoSyslog overrides the PID from the original message with the PID of the syslog-ng process.

version()

Description: The Elasticsearch version that AxoSyslog reports to the clients in the version handshake. The Elastic clients refuse a server whose major and minor version is older than their own, unless you enable the allow_older_versions option on the client side. Set this option to the version your clients expect.