Send logs, metrics, and traces to OpenTelemetry
Starting with version 4.3.0, AxoSyslog can send logs, metrics, and traces to OpenTelemetry over the OpenTelemetry Protocol (OTLP/gRPC).
The only required parameter is the url()
of the destination server, which includes the port number as well.
Example: Forwarding OpenTelemetry data
log otel_forward_mode_alts {
source {
opentelemetry(
port(12345)
auth(alts())
);
};
destination {
opentelemetry(
url("my-otel-server:12345")
auth(alts())
);
};
};
Example: Sending log messages to OpenTelemetry
The following example receives syslog messages and forwards them as OpenTelemetry logs.
log non_otel_to_otel_tls {
source {
network(
port(12346)
);
};
destination {
opentelemetry(
url("my-otel-server:12346")
auth(
tls(
ca-file("/path/to/ca.pem")
key-file("/path/to/key.pem")
cert-file("/path/to/cert.pem")
)
)
);
};
};
auth()
You can set authentication in the auth()
option of the driver. By default, authentication is disabled (auth(insecure())
).
The following authentication methods are available in the auth()
block:
adc()
Application Default Credentials (ADC). This authentication method is only available for destinations.
alts()
Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account()
option, where you can list service accounts to match against when authenticating the server.
source {
opentelemetry(
port(4317)
auth(alts())
);
};
destination {
loki(
port(12345)
auth(alts())
);
};
source {
syslog-ng-otlp(
port(4317)
auth(alts())
);
};
insecure()
This is the default method, authentication is disabled (auth(insecure())
).
tls()
tls()
accepts the key-file()
, cert-file()
, ca-file()
and peer-verify()
(possible values:
required-trusted
, required-untrusted
, optional-trusted
and optional-untrusted
) options.
destination {
opentelemetry(
url("your-otel-server:12346")
auth(
tls(
ca-file("/path/to/ca.pem")
key-file("/path/to/key.pem")
cert-file("/path/to/cert.pem")
)
)
);
};
destination {
loki(
url("your-loki-server:12346")
auth(
tls(
ca-file("/path/to/ca.pem")
key-file("/path/to/key.pem")
cert-file("/path/to/cert.pem")
)
)
);
};
destination {
syslog-ng-otlp(
url("your-otel-server:12346")
auth(
tls(
ca-file("/path/to/ca.pem")
key-file("/path/to/key.pem")
cert-file("/path/to/cert.pem")
)
)
);
};
Note:
tls(peer-verify())
is not available for theopentelemetry()
andloki()
destination.- The gRPC-based drivers (
opentelemetry()
andloki()
) have a differenttls()
block implementation from thenetwork()
orhttp()
drivers. Most features are the same.
batch-bytes()
Accepted values: | number [bytes] |
Default: | 4MB |
Available in AxoSyslog version 4.6 and later.
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines()
option. The batch might be at most 1 message larger than the set limit.
Note that if the batch-timeout()
option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout()
expires, or the batch reaches the limit set in batch-bytes()
.
OTLP has a default 4 MiB batch limit, therefore the default value for batch-bytes()
is 4 MB, which is a bit below 4 MiB.
The batch size is calculated before compression, which is the same as the limit is calculated on the server.
batch-lines()
Type: | number |
Default: | 1 |
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines()
to 100, AxoSyslog waits for 100 messages.
If the batch-timeout()
option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines()
number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout()
option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout()
expires, or the batch reaches the limit set in batch-lines()
.
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size()
option of the source must be higher than the batch-lines()
*workers()
of the destination. Otherwise, the size of the batches cannot reach the batch-lines()
limit.
batch-timeout()
Type: | time in milliseconds |
Default: | -1 (disabled) |
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout()
milliseconds.
channel-args()
Type: | arrow list |
Default: | - |
Description: The channel-args()
option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:
channel-args(
"grpc.loadreporting" => 1
"grpc.minimal_stack" => 0
)
compression()
Type: | boolean |
Default: | no |
Available in AxoSyslog version 4.5.0 and later.
Description: Enables compression in gRPC requests. Although gRPC supports various compression methods, currently only deflate is supported (which is basically the same as gzip).
headers()
Type: | arrow list |
Default: | empty |
Available in AxoSyslog 4.8 and later.
Description: Adds custom gRPC headers to each RPC call. Currently only static header names and values are supported.
headers(
"organization" => "Axoflow"
"stream-name" => "axo-stream"
)
workers()
Type: | integer |
Default: | 1 |
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Hazard of data loss.
When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.