ehttp: Receive logs over HTTP

Available in AxoSyslog 4.28 and later.

The ehttp() source receives log messages over HTTP or HTTPS. Clients send the messages in the body of an HTTP request, and AxoSyslog answers every accepted request.

Unlike the webhook() source, which is a Python-based configuration snippet, ehttp() is implemented natively in AxoSyslog. It is intended to replace webhook() in the long run.

Example: Receiving logs over HTTP

The following source accepts JSON payloads on port 8080, requires an authentication token, and answers the requests with a custom response body:

source s_http {
  ehttp(
    port(8080)
    mode("json")
    auth-token("Bearer s3cr3t")
    response-body('{"status": "received"}')
    flags(no-parse)
  );
};

To test the source, you can use curl on the host where AxoSyslog is running:

curl -X POST -H "Authorization: Bearer s3cr3t" --data '{"message": "hello"}' http://127.0.0.1:8080/

Note the flags(no-parse) option in the example: by default, AxoSyslog parses every extracted message as a syslog message. Use flags(no-parse) if the payload is not in syslog format, so that the entire payload becomes the ${MESSAGE} of the log message.

Compressed requests

The ehttp() source accepts gzip and deflate compressed request bodies, and decompresses them based on the Content-Encoding header of the request. The max-request-size() limit applies to the decompressed body as well.

Responses and error handling

The ehttp() source answers the requests of the clients as follows:

• The request is accepted: 200 OK, with the body set in response-body(), or the OK status line if response-body() is not set.
• The auth-token() option is set, but the Authorization header of the request is missing or doesn’t match: 401 Unauthorized
• The request is larger than max-request-size(), or its decompressed body is: 413 Payload Too Large
• The Content-Encoding of the request is not supported: 415 Unsupported Media Type
• The request is not valid HTTP, or its compressed body is corrupt: 400 Bad Request

If the body of the request cannot be parsed in the configured mode(), AxoSyslog logs a warning that includes the parse error and the offset where parsing failed, but still answers the request with 200 OK.

ehttp() source options

The ehttp() source has the following options.

auth-token()

Description: Requires the clients to authenticate themselves with a token. When set, AxoSyslog accepts only the requests whose Authorization header exactly matches the value of this option, and rejects every other request with 401 Unauthorized.

Include the authentication scheme in the value, because AxoSyslog compares the whole header:

auth-token("Bearer s3cr3t")

Since the token is sent in clear text, use this option only over HTTPS, that is, together with transport(tls).

chain-hostnames()

Description: Enable or disable the chained hostname format. For details, see the chain-hostnames() global option.

check-hostname()

Description:

When receiving messages, AxoSyslog can check whether the hostname contains valid characters.

Valid characters are:

• alphanumeric characters (A-Z, a-z, 0-9)
• the dash (-) and underscore (_) characters
• the dot (.) and the colon (:) characters
• the @ and slash (/)

If the hostname contains invalid characters, AxoSyslog sets the syslog.invalid_hostname tag for the message, and doesn’t parse the ${HOST} field from the message.

Uses the value of the global option if not specified.

check-program()

Description:

Available in version 4.10 and later.

If the check-program flag is enabled, AxoSyslog validates the ${PROGRAM} field for RFC3164-formatted messages. Valid program names meet the following criteria:

• Contain only these characters: [a-zA-Z0-9-_/().]
• Include at least one alphabetical character.

If the program name fails validation, it’s considered to be part of the log message.

default-facility()

Description: This parameter assigns a facility value to the messages received from the source if the message does not specify one. The default is user (kern for kernel sources).

default-severity()

Description: Assigns a default severity to messages received from this source when the message doesn’t specify one. Accepts the standard syslog severity keywords (for example, emerg, alert, crit, err, warning, notice, info, debug). Combined with default-facility() to compute the message priority.

dns-cache()

Description: Enable or disable DNS cache usage.

flags()

Description: Specifies the log parsing options of the source. For example:

source { network(flags(check-hostname, check-program)); };

assume-utf8

The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.

check-hostname

When receiving messages, AxoSyslog can check whether the hostname contains valid characters.

Valid characters are:

• alphanumeric characters (A-Z, a-z, 0-9)
• the dash (-) and underscore (_) characters
• the dot (.) and the colon (:) characters
• the @ and slash (/)

If the hostname contains invalid characters, AxoSyslog sets the syslog.invalid_hostname tag for the message, and doesn’t parse the ${HOST} field from the message.

check-program

Available in version 4.10 and later.

If the check-program flag is enabled, AxoSyslog validates the ${PROGRAM} field for RFC3164-formatted messages. Valid program names meet the following criteria:

• Contain only these characters: [a-zA-Z0-9-_/().]
• Include at least one alphabetical character.

If the program name fails validation, it’s considered to be part of the log message.

You can also enable this behavior globally using the check-program global option.

dont-store-legacy-msghdr

By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

empty-lines

Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.

exit-on-eof

If this flag is set on a source, AxoSyslog stops when an EOF (end of file) is received. Available in version 4.9 and later.

expect-hostname

If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

guess-timezone

Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

ignore-aux-data

The ignore-aux-data flag stops AxoSyslog from collecting the auxiliary data that the transport layer can attach to each incoming message, and from setting the related name-value pairs. Auxiliary data includes:

• The UNIX credentials of the sender process on UNIX domain sockets: ${.unix.pid}, ${.unix.uid}, ${.unix.gid}, ${.unix.cmdline}, ${.unix.exe}, ${.audit.auid}, and ${.audit.ses}. For details, see UNIX credentials and other metadata.
• The fields of the peer certificate on TLS-encrypted transports: ${.tls.x509_cn}, ${.tls.x509_o}, ${.tls.x509_ou}, and ${.tls.x509_fp}.
• The receive timestamp and the peer address that the transport layer provides for the message.

Use this flag if you do not need this metadata and want to avoid the cost of collecting it. Gathering UNIX credentials is especially expensive, because AxoSyslog reads the /proc filesystem for every message.

kernel

The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

no-header

The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.

Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.

The following example illustrates using the no-header flag with the syslog-parser() parser:

parser p_syslog {
    syslog-parser(
    flags(no-header)
    );
};

no-hostname

Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

source s_dell {
    network(
        port(2000)
        flags(no-hostname)
    );
};

no-multi-line

The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.

no-parse

By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

sanitize-utf8

When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.

Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

store-legacy-msghdr

By default, AxoSyslog stores the original incoming header of the log message, so this flag is active. To disable it, use the dont-store-legacy-msghdr flag.

store-raw-message

Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.

syslog-protocol

The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

validate-utf8

The validate-utf8 flag enables encoding-verification for messages.

Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.

For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

format()

Description: The message format parser that AxoSyslog uses on the extracted messages. In practice the only meaningful value is syslog, which is also the default. To skip parsing entirely, use flags(no-parse) instead.

host-override()

Description: Replaces the ${HOST} part of the message with the parameter string.

idle-timeout()

Available in AxoSyslog 4.9 and later.

If set, AxoSyslog closes the client connection if no data is received for the specified amount of time (in seconds).

internal()

Description: Marks this pipeline element as internal. Elements marked as internal() are treated as an implementation detail, so for example statistics of the given pipe are available only on higher stats level. This option is mainly useful for developers or when writing SCL blocks and integrations.

ip() or localip()

Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.

ip-freebind()

Description: Linux-only. Enables the IP_FREEBIND socket option, allowing AxoSyslog to bind to a non-local IP address. Useful when the address isn’t yet configured on any interface (for example, during failover scenarios) so the source can listen before the address becomes available. Has no effect on platforms that don’t support IP_FREEBIND.

ip-protocol()

Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).

Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).

For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.

source s_network_tcp { network( transport("tcp") ip("::") ip-protocol(6) port(601) ); };

ip-tos()

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()

Description: Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Description: Specifies whether connections to sources should be closed when AxoSyslog is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the AxoSyslog connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep-hostname()

Description: Enable or disable hostname rewriting.

• If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.

• If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

keep-timestamp()

Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

listen-backlog()

Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:

   source s_network {
        network(
            ip("192.168.1.1")
            transport("tcp")
            listen-backlog(2048)
            );
    };

log-fetch-limit()

Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()

Description: The size of the source window, that is, the maximum number of in-flight messages the source permits before flow control is enforced. This applies only when flags(flow-control) is set. For details, see Managing incoming and outgoing messages with flow-control.

log-msg-size()

Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).

For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.

For details on how encoding affects the size of the message, see Message size and encoding.

You can use human-readable units when setting configuration options. For details, seeNotes about the configuration syntax.

Uses the value of the global option if not specified.

log-prefix() (DEPRECATED)

This option is deprecated. Use program-override instead.

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.

max-connections()

Description: The maximum number of simultaneous client connections that the source accepts.

max-request-size()

Description: The maximum size of a single HTTP request, in bytes. AxoSyslog rejects larger requests with 413 Payload Too Large and logs an error.

For compressed requests, this limit applies to the decompressed body as well.

mode()

Description: Determines how AxoSyslog splits the body of the request into log messages.

• auto: If the body starts with { or [ and parses as JSON, AxoSyslog handles it as json, otherwise as line-separated.
• json: The body is JSON. If it is an array, every element becomes a separate log message. If it is a sequence of concatenated JSON objects (like the payload of the Splunk HTTP Event Collector), every object becomes a separate log message.
• line-separated (or jsonl): AxoSyslog splits the body on newline characters, and creates a log message from every non-empty line.
• single: The entire body becomes a single log message.

The value is case-insensitive, and must be specified as a string, for example, mode("json").

normalize-hostnames()

Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.

persist-name()

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

port() or localport()

Description: The port number to bind to. Make sure to enable this port on the firewall of the AxoSyslog host.

program-override()

Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.

read-old-records()

Description: If set to yes, AxoSyslog will start reading the records from the beginning of the file/journal, if the file/journal has not been read yet. If set to no, AxoSyslog will read only the new records. If the source has a state in the persist file, this option will have no effect.

response-body()

Description: The body that AxoSyslog sends in the 200 OK response. If you don’t set this option, the response body is the OK status line.

Because the value is a template, you can include the macros of the received message in the response. AxoSyslog evaluates the template using the first log message extracted from the request, for example:

response-body('{"status": "received", "host": "${HOST}"}')

sdata-prefix()

Available in AxoSyslog 4.1 and later.

Description: Adds a specific string before the names of the parsed SDATA fields to store the name-value pairs created from the SDATA fields separately. Note that unless the value of sdata-prefix starts with .SDATA., using this option excludes the parsed fields from the sdata and rfc5424 scopes of the value pairs.

so-broadcast()

Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.

so-keepalive()

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so-rcvbuf()

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so-sndbuf()

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()

Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.

tcp-keepalive-intvl()

Description: Specifies the interval (number of seconds) between subsequent keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

tcp-keepalive-probes()

Description: Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

tcp-keepalive-time()

Description: Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.

Available in AxoSyslog version 3.4 and later.

time-zone()

Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

tls()

Description: The TLS settings of the source, for example, the certificate and the private key that AxoSyslog shows to the clients. Setting transport(tls) requires a tls() block. For details on the individual options, see TLS options.

tls(
    key-file("/etc/syslog-ng/server.key")
    cert-file("/etc/syslog-ng/server.crt")
)

The following TLS options are available: ca-dir(), cert-file(), cipher-suite(), crl-dir(), dhparam-file(), ecdh-curve-list(), key-file(), peer-verify(), pkcs12-file(), ssl-options(), trusted-dn(), and trusted-keys().

transport()

Description: Sets whether the source accepts plain HTTP (tcp) or HTTPS (tls) connections. Setting transport(tls) requires a tls() block, and changes the default port to 443.

use-dns()

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from a file (for example, from /etc/hosts) set using the dns-cache-hosts() global option. The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

For details on configuring local DNS resolution, see Using name resolution in syslog-ng.

use-fqdn()

Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.

use-syslogng-pid()

Description: When enabled, AxoSyslog overrides the PID from the original message with the PID of the syslog-ng process.