Send logs, metrics, and traces to OpenTelemetry

Starting with version 4.3.0, AxoSyslog can send logs, metrics, and traces to OpenTelemetry over the OpenTelemetry Protocol (OTLP/gRPC).

The only required parameter is the url() of the destination server, which includes the port number as well.

Example: Forwarding OpenTelemetry data

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

Example: Sending log messages to OpenTelemetry

The following example receives syslog messages and forwards them as OpenTelemetry logs.

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

auth()

You can set authentication in the auth() option of the driver. By default, authentication is disabled (auth(insecure())).

The following authentication methods are available in the auth() block:

adc()

Application Default Credentials (ADC). This authentication method is only available for destinations.

alts()

Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.

• Driver:
• opentelemetry()
• loki()
• syslog-ng-otlp()

source {
    opentelemetry(
      port(4317)
      auth(alts())
    );
  };

destination {
    loki(
      port(12345)
      auth(alts())
    );
  };

source {
    syslog-ng-otlp(
      port(4317)
      auth(alts())
    );
  };

insecure()

This is the default method, authentication is disabled (auth(insecure())).

tls()

tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.

• Driver:
• opentelemetry()
• loki()
• syslog-ng-otlp()

destination {
    opentelemetry(
      url("your-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };

destination {
    loki(
      url("your-loki-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };

destination {
    syslog-ng-otlp(
      url("your-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };

Note:

• tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
• The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.

batch-bytes()

Available in AxoSyslog version 4.6 and later.

Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option. The batch might be at most 1 message larger than the set limit.

Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().

OTLP has a default 4 MiB batch limit, therefore the default value for batch-bytes() is 4 MB, which is a bit below 4 MiB.

The batch size is calculated before compression, which is the same as the limit is calculated on the server.

batch-lines()

Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.

For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.

If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.

Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().

For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.

batch-timeout()

Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.

channel-args()

Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:

    channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
    )

compression()

Available in AxoSyslog version 4.5.0 and later.

Description: Enables compression in gRPC requests. Although gRPC supports various compression methods, currently only deflate is supported (which is basically the same as gzip).

workers()

Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.