elasticsearch-http: Send messages to Elasticsearch HTTP Bulk API

Version 3.21 of AxoSyslog can directly post log messages to an Elasticsearch deployment using the Elasticsearch Bulk API over the HTTP and Secure HTTP (HTTPS) protocols.

HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format.

Prerequisites

• Install the axosyslog-scl package on Debian/Ubuntu. On RHEL and compatible distributions, the SCL files are part of the axosyslog base package.

  Your configuration must also contain @include "scl.conf", which the default configuration file already does. If the SCL files are missing, AxoSyslog fails to start with an unexpected LL_IDENTIFIER error.

  @include "scl.conf"

  The elasticsearch-http() driver is a reusable configuration snippet (SCL). For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find its source in scl/elasticsearch/elastic-http.conf on GitHub.

• This feature requires a separate module. Install the axosyslog-mod-http package on Debian/Ubuntu, or the axosyslog-http package on RHEL and compatible distributions. If the module isn’t installed, AxoSyslog fails to start with an unexpected LL_IDENTIFIER error.

Declaration:

   d_elasticsearch_http {
        elasticsearch-http(
            index("<elasticsearch-index-to-store-messages>")
            url("https://your-elasticsearch-server1:9200/_bulk")
            type("<type-of-the-index>")
        );
    };

Use an empty string to omit the type from the index: type(""). For example, you need to do that when using Elasticsearch 7 or newer, and you use a mapping in Elasticsearch to modify the type of the data.

You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.

Alternatively, you can leave the HTTP as-is, in which case the driver leaves the default http_proxy and https_proxy environment variables unmodified.

For more detailed information about these environment variables, see the libcurl documentation.

Example: Sending log data to Elasticsearch

The following example defines an elasticsearch-http() destination, with only the required options.

   destination d_elasticsearch_http {
        elasticsearch-http(
            index("<name-of-the-index>")
            type("<type-of-the-index>")
            url("http://my-elastic-server:9200/_bulk")
        );
    };
    
    
    log {
        source(s_file);
        destination(d_elasticsearch_http);
        flags(flow-control);
    };

The following example uses mutually-authenticated HTTPS connection, templated index, and also sets the type() and some other options.

   destination d_elasticsearch_https {
        elasticsearch-http(
            url("https://node01.example.com:9200/_bulk")
            index("test-${YEAR}${MONTH}${DAY}")
            time-zone("UTC")
            type("test")
            workers(4)
            batch-lines(16)
            timeout(10)
            tls(
                ca-file("ca.pem")
                cert-file("syslog_ng.crt.pem")
                key-file("syslog_ng.key.pem")
                peer-verify(yes)
            )
        );
    };