AxoSyslog is a cloud-native syslog-ng distribution by Axoflow. If you want to try running syslog-ng in a container, or use it as a log collector in Kubernetes, try AxoSyslog!
In compliance with the above license, syslog-ng is used only to refer to the binary of the application, or in filenames, pathnames, and similar technical terms. For all other references, we use the AxoSyslog name.
New files and modifications compared to the syslog-ng Open Source Edition documentation are under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks.
The Axoflow and AxoSyslog trademarks and logos are trademarks of Axoflow Inc. All other trademarks are property of their respective owners.
This documentation is maintained by Axoflow, and continually updated for new releases.
Acknowledgements
We would like to express our gratitude to everyone involved with the project, including the syslog-ng developers, advocates, and documentation maintainers who worked at BalaBit or are still working at One Identity, or who helped the project as external contributors, users, or as part of the community.
Feedback
Any feedback is greatly appreciated, especially on what else this document should cover. General comments, errors found in the text, and any suggestions about how to improve the documentation are also welcome as GitHub issues, or on the Axoflow Discord server.
Support
In case you need help with any of the AxoSyslog projects, or directly with syslog-ng, you have several ways to contact us:
We also provide consulting and professional services for logging and observability related projects. Contact us if you need our help!
1 - What's new
This page is a changelog that collects the major changes and additions to this documentation. (If you want to know the details about why we have separate documentation for AxoSyslog, our syslog-ng distribution and how it relates to the syslog-ng documentation, read our syslog-ng documentation and similarities with AxoSyslog Core blog post.)
This chapter introduces the AxoSyslog application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure.
What AxoSyslog is
The AxoSyslog application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging and observability solutions and telemetry pipelines, supporting on-premises, cloud-native, and hybrid environments.
Among others, AxoSyslog allows you the following.
Secure and reliable data transfer
The AxoSyslog application enables you to send log messages and other observability data from your hosts to remote servers using the latest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transfer your log messages and observability data using reliable protocols to ensure that no messages are lost.
Disk-based message buffering
To minimize the risk of losing important log messages, the AxoSyslog application can store messages on the local hard disk if the central log server or the network connection becomes unavailable. The AxoSyslog application automatically sends the stored messages to the server when the connection is reestablished, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if AxoSyslog is restarted.
Secure logging using TLS
Log messages may contain sensitive information that should not be accessed by third parties. Therefore, AxoSyslog supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.
Flexible data extraction and processing
Most log messages are inherently unstructured, which makes them difficult to process. To overcome this problem, AxoSyslog comes with a huge set of built-in parsers, which you can combine to build very complex things.
Filter and classify
The AxoSyslog application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. You can create directories, files, and database tables dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.
Parse and rewrite
The AxoSyslog application can segment log messages to named fields or columns, and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.
To get the most information out of your log data, AxoSyslog allows you to correlate multiple log messages and aggregate the extracted information into a single message. You can also use external information to enrich your log data.
Big data clusters
The log data that your organization has to process, store, and review increases daily, so many organizations use big data solutions for their logs. To accommodate this huge amount of data, AxoSyslog natively supports storing log messages and observability data in destinations like HDFS files, Elasticsearch/OpenSearch, and cloud storage solutions like Google Pub/Sub and Amazon S3.
Message queue support
Large organizations increasingly rely on queuing infrastructure to transfer their data. For that purpose, AxoSyslog supports Apache Kafka, the Advanced Message Queuing Protocol (AMQP), and the Simple Text Oriented Messaging Protocol (STOMP).
SQL, NoSQL, and monitoring
Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The AxoSyslog application supports the following databases: MongoDB, MSSQL, MySQL, Oracle, PostgreSQL, and SQLite.
AxoSyslog also allows you to extract the information you need from your log data, and directly send it to your Graphite, Redis, or Riemann monitoring system.
syslog protocol standards
AxoSyslog not only supports legacy BSD syslog (RFC3164) and the enhanced RFC5424 protocols but also JavaScript Object Notation (JSON) and journald message formats.
Heterogeneous on-premises and cloud-native environments
The AxoSyslog application is the ideal choice to collect logs and observability data in massively heterogeneous environments that use several different operating systems and hardware platforms, including Linux, Unix, BSD, Sun Solaris, HP-UX, MacOS, AIX. You can also run it in containerized environments.
IPv4 and IPv6 support
The AxoSyslog application can operate in both IPv4 and IPv6 network environments, and can receive and send messages to both types of networks.
What AxoSyslog is not
The AxoSyslog application is not log analysis software. It can filter log messages and select only the ones matching certain criteria. It can even convert the messages and restructure them to a predefined format, or parse the messages and segment them into different fields. But AxoSyslog cannot interpret and analyze the meaning behind the messages, or recognize patterns in the occurrence of different messages.
Why do you need AxoSyslog
Log messages contain information about the events happening on the hosts. Monitoring system events is essential for security and system health monitoring reasons.
The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility, and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The AxoSyslog application helps you to select only the really interesting messages, and forward them to a central server.
Company policies or other regulations often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process.
Who uses AxoSyslog
The AxoSyslog application is used worldwide by companies and institutions who collect and manage the logs of several hosts, and want to store them in a centralized, organized way. Using AxoSyslog is particularly advantageous for:
Internet Service Providers
Financial institutions and companies requiring policy compliance
Server, web, and application hosting companies
Datacenters
Wide area network (WAN) operators
Server farm administrators.
2.1 - Supported platforms
The AxoSyslog application is highly portable and is known to run on a wide range of hardware architectures (x86, x86_64, SUN Sparc, PowerPC 32 and 64, Alpha) and operating systems, including Linux, BSD, Solaris, IBM AIX, HP-UX, Mac OS X, Cygwin, and others.
This chapter provides a very brief introduction into configuring the AxoSyslog application. For details on the format of the configuration file and how to configure sources, destinations, and other features, refer to the subsequent chapters.
To configure AxoSyslog as a relay that receives log messages from client hosts and forwards them to a central logserver, see Configuring AxoSyslog on server hosts.
To configure AxoSyslog on a client host, complete the following steps.
Steps:
Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.
Configure the local sources to collect the log messages of the host. Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.
To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.
Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog collector Helm chart.
Add sources to collect the messages from your log files. File sources look like this:
Many applications send log messages to logfiles by default (for example, the Roundcube webmail client, or the ProFTPD FTP server), but can be configured to send them to syslog instead. If possible, it is recommended to reconfigure the application that way.
Note
The default configuration file of AxoSyslog collects platform-specific log messages and the internal log messages of AxoSyslog.
source s_local { system(); internal();};
Create a network destination that points directly to the AxoSyslog server, or to a local relay. The network destination greatly depends on the protocol that your log server or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:
Create a log statement connecting the local sources to the file destination.
Note
The default configuration of AxoSyslog has only one log statement:
log { source(s_local); destination(d_local);};
Set filters, macros and other features and options (for example, TLS encryption) as necessary.
Example: The default configuration file of AxoSyslog
The following is the default configuration file of AxoSyslog4.5.0. It collects local log messages and the log messages of AxoSyslog and saves them in the /var/log/messages file.
To configure AxoSyslog on a server host, complete the following steps.
Steps:
Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.
Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.
To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.
Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog collector Helm chart.
Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:
Starting with AxoSyslog version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164) and IETF-syslog (RFC 5424-26) messages.
Create local destinations that will store the log messages, for example, file- or program destinations. The default configuration of AxoSyslog places the collected messages into the /var/log/messages file:
Set filters, options (for example, TLS encryption) and other advanced features as necessary.
Note
By default, the AxoSyslog server treats the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the AxoSyslog server, use the keep-hostname(yes) option both on the AxoSyslog relay and the AxoSyslog server. This option can be set individually for every source if needed.
If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.
Example: A simple configuration for servers
The following is a simple configuration file for AxoSyslog that collects incoming log messages and stores them in a text file.
This section describes how to configure AxoSyslog as a relay.
3.3.1 - Configuring AxoSyslog on relay hosts
To configure AxoSyslog on a relay host, complete the following steps:
Install the AxoSyslog application on the host. For details on installing AxoSyslog on specific operating systems, see Install AxoSyslog.
Configure the network sources that collect the log messages sent by the clients.
Create a network destination that points to the AxoSyslog server.
Create a log statement connecting the network sources to the AxoSyslog server.
Configure the local sources that collect the log messages of the relay host.
Create a log statement connecting the local sources to the AxoSyslog server.
Enable the keep-hostname() and disable the chain-hostnames() options. (For details on how these options work, see Global options.)
Note
It is recommended to use these options on your AxoSyslog server as well.
Set filters and options (for example, TLS encryption) as necessary.
Note
By default, the AxoSyslog server treats the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the AxoSyslog server, use the keep-hostname(yes) option both on the AxoSyslog relay and the AxoSyslog server. This option can be set individually for every source if needed.
If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.
Example: A simple configuration for relays
The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.
Depending on your exact needs about relaying log messages, there are many scenarios and AxoSyslog options that influence how the log message will look like on the logserver. Some of the most common cases are summarized in the following example:
Consider the following example: client-host > relay > server, where the IP address of client-host is 192.168.1.2. The client-host device sends a syslog message to relay. Depending on the settings of relay, the following can happen.
By default, the keep-hostname() option is disabled, so relay writes the IP address of the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discarding any IP address or hostname that was originally in the message.
If the keep-hostname() option is enabled on relay, but name resolution is disabled (the use-dns() option is set to no), relay uses the HOST field of the message as-is, which is probably 192.168.1.2.
To resolve the 192.168.1.2 IP address to a hostname on relay using a DNS server, use the keep-hostname(no) and use-dns(yes) options. If the DNS server is properly configured and reverse DNS lookup is available for the 192.168.1.2 address, AxoSyslog will rewrite the HOST field of the log message to client-host.
Note
It is also possible to resolve IP addresses locally, without relying on the DNS server. For details on local name resolution, see Resolving hostnames locally.
The above points apply to the AxoSyslog server (server) as well, so if relay is configured properly, use the keep-hostname(yes) option on server to retain the proper HOST field. Setting keep-hostname(no) on server would result in AxoSyslog rewriting the HOST field to the address of the host that sent the message to server, which is relay in this case.
If you cannot or do not want to resolve the 192.168.1.2 IP address on relay, but want to store your log messages on server using the IP address of the original host (that is, client-host), you can enable the spoof-source() option on relay. However, spoof-source() works only under the following conditions:
The syslog-ng binary has been compiled with the --enable-spoof-source option.
The log messages are sent using the highly unreliable UDP transport protocol. (Extremely unrecommended.)
3.4 - Sending Kubernetes logs to OpenSearch
The following tutorial shows you how to install AxoSyslog on Kubernetes, deploy OpenSearch to the same cluster, and send Kubernetes logs to OpenSearch.
Prerequisites
You need:
A Kubernetes cluster. We used minikube with docker driver and Helm. We used a Ubuntu 22.04 (amd64) machine, but it should work on any system that can run minikube (2 CPUs, 2GB of free memory, 20GB of free disk space).
The OpenSearch service needs a large mmap count setting, so set it to at least 262144, for example:
If you don’t already have an application that generates logs deployed to the Kubernetes cluster, install kube-logging/log-generator to generate sample logs. Complete the following steps.
"kube-logging" has been added to your repositories
Update your repositories.
helm repo update
The output should look like:
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kube-logging" chart repository
Update Complete. ⎈Happy Helming!⎈
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "opensearch" chart repository
Update Complete. ⎈Happy Helming!⎈
Install OpenSearch. This step can take a few minutes.
NAME: opensearch-dashboards-1684695728
LAST DEPLOYED: Sun May 21 21:02:09 2023NAMESPACE: default
STATUS: deployed
REVISION: 1TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
exportPOD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch-dashboards-1684695728" -o jsonpath="{.items[0].metadata.name}")exportCONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")echo"Visit http://127.0.0.1:8080 to use your application" kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
Now you should have 5 pods. Check that they exist:
kubectl get pods
The output should look like:
NAME READY STATUS RESTARTS AGE
log-generator-1681984863-5946c559b9-ftrrn 1/1 Running 0 3m39s
opensearch-cluster-master-0 1/1 Running 0 81s
opensearch-cluster-master-1 1/1 Running 0 81s
opensearch-cluster-master-2 1/1 Running 0 81s
opensearch-dashboards-1681999620-59f64f98f7-bjwwh 1/1 Running 0 44s
Forward the 5601 port of the OpenSearch Dashboards service (replace the name of the pod with your pod).
Forwarding from 127.0.0.1:8080 -> 5601Forwarding from [::1]:8080 -> 5601
Log in to the dashboard at http://localhost:8080 with admin/admin. You will soon create an Index Pattern here, but first you have to send some logs from syslog-ng.
Create a YAML file (called axoflow-demo.yaml in the examples) to configure the collector.
config:sources:kubernetes:# Collect kubernetes logsenabled:truedestinations:# Send logs to OpenSearchopensearch:- address:"opensearch-cluster-master"index:"test-axoflow-index"user:"admin"password:"admin"tls:# Do not validate the server's TLS certificate.peerVerify:false# Send the syslog fields + the metadata from .k8s.* in JSON formattemplate:"$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))"
Check how the syslog-ng.conf file looks with your custom values:
Open OpenSearch dashboard at http://localhost:8080/app/management/opensearch-dashboards/.
Create an Index Pattern called test-axoflow-index: http://localhost:8080/app/management/opensearch-dashboards/indexPatterns. At Step 2, set the Time field to @timestamp.
Now you can see your logs on the Discover view at http://localhost:8080/app/discover. Opening the detailed view for a log entry shows you the fields sent to OpenSearch.
3.5 - Managing and checking the syslog-ng service on Linux
This section describes how to start, stop and check the status of AxoSyslog service on Linux.
Starting AxoSyslog
To start AxoSyslog, execute the following command as root. For example:
systemctl start syslog-ng
If the service starts successfully, no output will be displayed.
Job for syslog-ng.service failed because the control process exited with error code. See `systemctl status syslog-ng.service` and `journalctl -xe` for details.
The internal logs of AxoSyslog contains informal, warning and error messages.
By default, AxoSyslog log messages (generated on the internal() source) are written to /var/log/messages.
Check the internal logs of AxoSyslog for any issue.
Message processing
The AxoSyslog application collects statistics about the number of processed messages on the different sources and destinations.
Note
When using syslog-ng-ctl stats, consider that while the output is generally consistent, there is no explicit ordering behind the command. Consequently, Axoflow does not recommend creating parsers that depend on a fix output order.
If needed, you can sort the output with an external application, for example, | sort.
Central statistics
To check the central statistics, execute the following command to see the number of received and queued (sent) messages by AxoSyslog.
You can install AxoSyslog on many platforms using the package manager and official repositories of the platform. For a list of third-party packages available for various Linux, UNIX, and other platforms, see syslog-ng Open Source Edition installation packages.
For instructions on compiling syslog-ng Open Source Edition from the source code, see the GitHub project page.
4.1 - Install AxoSyslog with Podman and systemd
This page shows you how to run AxoSyslog as a systemd service using podman.
Edit the unit file as needed for your environment.
We recommend using the default mount points:
Purpose
On the host
In the container
Disk-buffer and persist files
/var/lib/syslog-ng
/var/lib/syslog-ng
syslog-ng configuration file
/opt/axosyslog/etc
/etc/syslog-ng
Output log files
/opt/axosyslog/var/log
/var/log
(Optional) Create an override.conf file to set custom environment values. This can be useful if you don’t want to modify /etc/containers/systemd/axosyslog.container. Run:
systemctl edit axosyslog
Later you can edit this file by running the previous command again.
Create the /opt/axosyslog/etc/syslog-ng.conf configuration file based on the following template.
With the following sample configuration file AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages file.
You can customize the configuration file according to your needs. For a few pointers, see Configuring AxoSyslog on server hosts and the rest of this guide.
Run the following commands to reload the systemd configuration and launch the axosyslog service. Though the systemctl commands are run as root, the container will run as the specified user if set appropriately in the unit file.
If there aren’t any errors, these commands don’t have any output.
Run the following command to verify that the service was properly started:
journalctl -b -u axosyslog | tail -100
The output should be similar to:
Feb 12 09:04:40 <your-hostname> systemd[1]: Starting AxoSyslog Container...
Feb 12 09:04:40 <your-hostname> podman[2783]: 2024-02-12 09:04:40.454665314 -0500 EST m=+0.167732500 system refresh
Feb 12 09:04:40 <your-hostname> axosyslog[2783]: Trying to pull ghcr.io/axoflow/axosyslog:latest...
Feb 12 09:04:40 <your-hostname> axosyslog[2783]: Pulling image //ghcr.io/axoflow/axosyslog:latest inside systemd: setting pull timeout to 5m0s
Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Getting image source signatures
Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:619be1103602d98e1963557998c954c892b3872986c27365e9f651f5bc27cab8
Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:b061f41886afb563aff2a5f731f3286ba54ea6f657ed3e282f5339a12a64c5ef
Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:1b8d965a650c6a05227bd5c549930c9898071e8e7abb26886d4169a99762de0a
Feb 12 09:04:41 <your-hostname> axosyslog[2783]: Copying blob sha256:b5b0ce6ebef193c4f909379188cfb59443e8a1809816fbb476074908b170b4d1
Feb 12 09:04:50 <your-hostname> axosyslog[2783]: Copying config sha256:c379d94ef2c5ec348dfb3a93eed9a19aed667c396008db85edc354c8f4f8cb6a
Feb 12 09:04:50 <your-hostname> axosyslog[2783]: Writing manifest to image destination
Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.422390687 -0500 EST m=+10.135457863 container create 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.url=https://axoflow.io/, org.opencontainers.image.source=https://github.com/axoflow/axosyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.title=AxoSyslog, org.opencontainers.image.vendor=Axoflow, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, maintainer=axoflow.io, org.opencontainers.image.licenses=GPL-3.0-only)Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.402626446 -0500 EST m=+10.115693622 image pull c379d94ef2c5ec348dfb3a93eed9a19aed667c396008db85edc354c8f4f8cb6a ghcr.io/axoflow/axosyslog:latest
Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.489925509 -0500 EST m=+10.202992695 container init 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.licenses=GPL-3.0-only, org.opencontainers.image.vendor=Axoflow, maintainer=axoflow.io, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.url=https://axoflow.io/, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.title=AxoSyslog, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, org.opencontainers.image.source=https://github.com/axoflow/axosyslog)Feb 12 09:04:50 <your-hostname> systemd[1]: Started AxoSyslog Container.
Feb 12 09:04:50 <your-hostname> podman[2783]: 2024-02-12 09:04:50.500050669 -0500 EST m=+10.213117845 container start 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f (image=ghcr.io/axoflow/axosyslog:latest, name=AxoSyslog, PODMAN_SYSTEMD_UNIT=axosyslog.service, org.opencontainers.image.source=https://github.com/axoflow/axosyslog, org.opencontainers.image.authors=Axoflow, org.opencontainers.image.description=A cloud-native distribution of syslog-ng by Axoflow, org.opencontainers.image.documentation=https://axoflow.com/docs/axosyslog/docs/, org.opencontainers.image.licenses=GPL-3.0-only, org.opencontainers.image.vendor=Axoflow, org.opencontainers.image.title=AxoSyslog, maintainer=axoflow.io, org.opencontainers.image.url=https://axoflow.io/)Feb 12 09:04:50 <your-hostname> axosyslog[2783]: 477c9f011684f767aae138a0f88602ff30a8c95a46d616bb3b95318ec3a4b79f
Feb 12 09:04:50 <your-hostname> AxoSyslog[2821]: [2024-02-12T14:04:50.806054] syslog-ng starting up;version='4.6.0'
Send a test message to the service:
echo'<5> localhost test: this is a test message'| nc localhost 514
Check that the test message has arrived into the log file:
less /opt/axosyslog/var/log/messages
The output should be similar to:
Feb 19 15:49:12 localhost test: this is a test message
Customize the configuration
To customize the configuration, edit the /opt/axosyslog/etc/syslog-ng.conf file on the host, then reload the service.
Managing the AxoSyslog systemd service
You can reload syslog-ng running in the container via systemctl. The following command reloads the syslog-ng.conf file, without stopping/starting syslog-ng itself.
sudo systemctl reload axosyslog
You can access syslog-ng-ctl from the host, for example by running:
If you use syslog-ng-ctl regularly, you can create the /opt/axosyslog/bin/syslog-ng-ctl file with the following content, make it executable, and add it to your path. That way running syslog-ng-ctl <command> will execute the command in the AxoSyslog container.
The traditional method of starting a service at boot (systemctl enable) is not supported for container services. To automatically start the AxoSyslog service, make sure that the following line is included in the unit file. (It is included in the sample template.)
NAME: axosyslog-collector-1683469360
LAST DEPLOYED: Sun May 7 16:22:40 2023NAMESPACE: default
STATUS: deployed
REVISION: 1TEST SUITE: None
NOTES:
1. Watch the axosyslog-collector-1683469360 container start.
$ kubectl get pods --namespace=default -l app=axosyslog-collector-1683469360 -w
Check that the pod is running.
kubectl get pods
The output should look like:
NAME READY STATUS RESTARTS AGE
axosyslog-collector-1683469360-tptfb 1/1 Running 0 28s
How to use disk-buffers in containers and Kubernetes
When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.
Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
In a container, mount the disk-buffer directory from the host, or store it on a local volume.
Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.
If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.
Uninstall
Tip: List all installed releases using helm list.
To uninstall a chart release, run:
helm delete <name-of-the-release-to-delete>
4.2.1 - Parameters of the AxoSyslog collector Helm chart
The following table lists the configurable parameters of the AxoSyslog collector chart and their default values. For details on installing the chart, see Install AxoSyslog with Helm.
Parameters for syslog-ng configuration
Parameter
Description
Default
config.raw
A complete syslog-ng configuration. If this parameter is set, all other parameters in the config section are ignored. For details on how to create a configuration for syslog-ng, see the AxoSyslog Core documentation.
Collect pod logs using the kubernetes() source. If disabled, the chart doesn’t configure any source. For the list of available sources, see the (/docs/axosyslog-core/chapter-sources/)
true
The following example uses the config.raw parameter to configure a custom destination:
Name of the OpenSearch index that stores the messages.
""
config.destination.opensearch.user
The username to use for authentication on the OpenSearch server, if not authenticating with a certificate.
""
config.destination.opensearch.password
The password to use for authentication on the OpenSearch server.
""
config.destination.opensearch.tls.CAFile
The CA certificate in PEM format to use when verifying the certificate of the server.
""
config.destination.opensearch.tls.CADir
A directory containing a set of trusted CA certificates in PEM format. The name of the files must be the 32-bit hash of the subject’s name. AxoSyslog collector verifies the certificate of the server using these CA certificates.
""
config.destination.opensearch.tls.Cert
Name of a file containing an X.509 certificate or a certificate chain in PEM format. AxoSyslog collector authenticates with this certificate on the server, with the private key set in the config.destination.opensearch.tls.Key field. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
""
config.destination.opensearch.tls.Key
Name of a file containing an unencrypted private key in PEM format. AxoSyslog collector authenticates with this key and the certificate set in the config.destination.opensearch.tls.Cert field.
""
config.destination.opensearch.tls.peerVerify
If true, AxoSyslog collector verifies the certificate of the server with the CA certificates set in config.destination.opensearch.tls.CAFile and config.destination.opensearch.tls.CADir.
You can also use it as a base image in your Dockerfile:
FROM ghcr.io/axoflow/axosyslog:latest
If you want to test a development version, you can use the nightly builds:
podman pull ghcr.io/axoflow/axosyslog:nightly
Note: These named packages are automatically updated when a new package is released. To install a specific version, run podman pull ghcr.io/axoflow/axosyslog:<version-number>, for example:
podman pull ghcr.io/axoflow/axosyslog:4.5.0
Customize the configuration
The AxoSyslog container image stores the configuration file at /etc/syslog-ng/syslog-ng.conf. By default, AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages and /var/log/messages-kv.log files using this configuration file from the syslog-ng repository.
To customize the configuration, create your own configuration file and override the file in the container image with it, for example:
podman run --rm --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest
How to use disk-buffers in containers and Kubernetes
When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.
Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
In a container, mount the disk-buffer directory from the host, or store it on a local volume.
Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.
If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.
Expose port to receive incoming traffic
To receive incoming network in a container, you must expose the port from the container where you want to receive the traffic to the host that’s running the container. Typically, this is only needed if you are running AxoSyslog as a relay or a server/aggregator.
By default, the AxoSyslog container images expose the ports commonly used to receive syslog traffic:
514/udp, typically used for RFC3164 (BSD-syslog) formatted traffic.
601/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic.
6514/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic over TLS.
To expose a specific port, use the --expose option when starting the container. Make sure to include the IP address of the host to make the port externally accessible.
For example, if you are receiving OpenTelemetry messages using the opentelemetry() source, expose the 4317 port:
podman run --rm --expose 127.0.0.1:4317:4317/tcp --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest
You can also use it as a base image in your Dockerfile:
FROM ghcr.io/axoflow/axosyslog:latest
If you want to test a development version, you can use the nightly builds:
docker pull ghcr.io/axoflow/axosyslog:nightly
Note: These named packages are automatically updated when a new package is released. To install a specific version, run docker pull ghcr.io/axoflow/axosyslog:<version-number>, for example:
docker pull ghcr.io/axoflow/axosyslog:4.5.0
Customize the configuration
The AxoSyslog container image stores the configuration file at /etc/syslog-ng/syslog-ng.conf. By default, AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages and /var/log/messages-kv.log files using this configuration file from the syslog-ng repository.
To customize the configuration, create your own configuration file and override the file in the container image with it, for example:
docker run --rm --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest
How to use disk-buffers in containers and Kubernetes
When you are running AxoSyslog in a container or in Kubernetes, and you want to use disk-buffers, there are some additional things to configure.
Make sure to mount the disk-buffer files and the persist file (by default, both are stored in /var/lib/syslog-ng) in a way they are not lost when the pod or container is restarted.
In Kubernetes, add a persistent volume to your pod and store the disk buffer files (/var/lib/syslog-ng) there.
In a container, mount the disk-buffer directory from the host, or store it on a local volume.
Use a reliable disk-buffer only if your storage is fast enough. For example, a low-speed persistent volume in Kubernetes can cause a significant performance degradation for AxoSyslog.
Use the latest available version of AxoSyslog, as many related improvements and performance improvements (for example, disk-buffer related metrics) are only supported in recent versions.
If you are using syslog-ng without disk-buffering configured, syslog-ng stores everything in memory, which results in great performance. If you enable disk-buffering, the performance decreases. Make sure to size your observability pipeline appropriately.
Expose port to receive incoming traffic
To receive incoming network in a container, you must expose the port from the container where you want to receive the traffic to the host that’s running the container. Typically, this is only needed if you are running AxoSyslog as a relay or a server/aggregator.
By default, the AxoSyslog container images expose the ports commonly used to receive syslog traffic:
514/udp, typically used for RFC3164 (BSD-syslog) formatted traffic.
601/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic.
6514/tcp, typically used for RFC5424 (IETF-syslog) formatted traffic over TLS.
To expose a specific port, use the --expose option when starting the container. Make sure to include the IP address of the host to make the port externally accessible.
For example, if you are receiving OpenTelemetry messages using the opentelemetry() source, expose the 4317 port:
docker run --rm --expose 127.0.0.1:4317:4317/tcp --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest
5 - The concepts of AxoSyslog
This chapter discusses the technical concepts of AxoSyslog.
5.1 - The philosophy of AxoSyslog
Typically, AxoSyslog is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices — called AxoSyslog clients — all run AxoSyslog, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote AxoSyslog server, which sorts and stores them.
5.2 - Logging with AxoSyslog
The AxoSyslog application reads incoming messages and forwards them to the selected destinations. The AxoSyslog application can receive messages from files, remote hosts, and other sources.
Log messages enter AxoSyslog in one of the defined sources, and are sent to one or more destinations.
Sources and destinations are independent objects, log paths define what AxoSyslog does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations: messages arriving from a source are sent to every destination listed in the log path. A log path defined in AxoSyslog is called a log statement.
Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, AxoSyslog sends only the messages satisfying the filter rules to the destinations set in the log path.
Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.
5.2.1 - The route of a log message in AxoSyslog
Purpose:
The following procedure illustrates the route of a log message from its source on the AxoSyslog client to its final destination on the central AxoSyslog server.
Steps:
A device or application sends a log message to a source on the AxoSyslog client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file.
The AxoSyslog client running on the web server reads the message from its /var/log/apache source.
The AxoSyslog client processes the first log statement that includes the /var/log/apache source.
The AxoSyslog client performs optional operations (message filtering, parsing, and rewriting) on the message, for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, AxoSyslog sends the message to the destinations set in the log statement, for example, to the remote AxoSyslog server.
Warning
Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
The AxoSyslog client processes the next log statement that includes the /var/log/apache source, repeating Steps 3-4.
The message sent by the AxoSyslog client arrives from a source set in the AxoSyslog server.
The AxoSyslog server reads the message from its source and processes the first log statement that includes that source.
The AxoSyslog server performs optional operations (message filtering, parsing, and rewriting) on the message, for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, AxoSyslog sends the message to the destinations set in the log statement.
Warning
Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.
The AxoSyslog server processes the next log statement, repeating Steps 7-9.
Note
The AxoSyslog application can stop reading messages from its sources if the destinations cannot process the sent messages. This feature is called flow-control and is detailed in Managing incoming and outgoing messages with flow-control.
5.3 - Modes of operation
The AxoSyslog application has three typical operation scenarios: Client, Server, and Relay.
5.3.1 - Client mode
In client mode, AxoSyslog collects the local logs generated by the host and forwards them through a network connection to the central AxoSyslog server or to a relay. Clients often also log the messages locally into files.
5.3.2 - Relay mode
In relay mode, AxoSyslog receives logs through the network from AxoSyslog clients and forwards them to the central AxoSyslog server using a network connection. Relays also log the messages from the relay host into a local file, or forward these messages to the central AxoSyslog server.
Example relay use cases
The relay collects log messages through the network and after processing, but without writing them on the disk for storage, forwards them to one or more remote destinations.
You can use a relay for many different use cases as described in the examples below.
UDP-only source devices
Most network devices send log messages over UDP. However, UDP does not guarantee that all packets are delivered, which makes UDP unreliable.
To ensure at least a best effort level of reliability, Axoflow recommends that you deploy a relay on the network, close to the source devices. With the most reliable hops between the source and the relay, you can minimize the risk of losing UDP packets. Once the packet arrives at the relay, AxoSyslog ensures that the messages are delivered to the central server in a reliable manner.
Too many source devices
Depending on the hardware and configuration, an average instance can usually handle the following number of concurrent connections:
If the maximum message rate is lower than 200,000 messages per second:
maximum ca. 5,000 TCP connections
maximum ca. 1,000 TLS connections
If the message rate is higher than 200,000 messages per second, contact Axoflow.
If you have more source devices, you must deploy a relay machine at least per 5,000 sources and batch up all the logs into a single TCP connection that connects the relay to the server. If TLS is used, deploy relays per 1,000 source devices.
Collecting logs from remote sites (especially over public WAN)
If you need to collect log messages from geographically remote sites or over public WAN, Axoflow recommends that you install at least a relay node per each remote site. The relay can be the last outgoing hop for all the messages of the remote site, which has several benefits:
Maintenance: You only need to change the configuration of the relay if you want to re-route the logs of some or all sources of the remote site. Also you do not need to change each source’s configuration one by one.
Security: If you trust your internal network, it is not necessary to hold encrypted connections within the LAN of the remote site as the messages can get to the relay without encryption. Messages must be sent in an encrypted way over the public WAN, and it is enough to hold only a single TCP/TLS connection between the sites, that is, between the remote relay and the central server. This eliminates the wasting of resources as holding several TLS connections directly from the clients is more costly than holding a single connection from the relay.
Reliability: You can set up a main disk-buffer on the relay. The main disk-buffer is only responsible for buffering all the logs of the remote site if the central AxoSyslog server is temporarily unavailable. It is easier to maintain this single main disk-buffer instead of setting disk-buffers on individual client machines.
Separation, distribution, and balancing of message processing tasks
Most Linux applications have their own human readable, but difficult to handle, log messages. Without parsing and normalization it is difficult to alert and report on these log messages. Many AxoSyslog users use the message parsing tools of AxoSyslog to normalize their different log messages. Just like normalization, filtering can also be resource-heavy, depending on what the filtering is based on. In this case, it might be inefficient to perform all the message processing tasks on the server as it can result in decreased overall performance.
It is a typical setup to deploy relays in front of the central server operating as a receiver front-end. Most resource-heavy tasks, for example, parsing, filtering, and so on, are performed on this receiver layer. As all resource-heavy tasks are performed on the relay, the central server behind it only needs to get the messages from the relay and write them into the final text-based or tamper-proof (logstore) format. Since you can run several relays, you can balance the resource-heavy tasks between more relays, and a single server behind the relays can still be fast enough to write all the messages on the disk.
Acting as a relay also depends on the functionality. A relay does not have to be a dedicated relay machine at all. For log collection, it can be one of the clients with a relay configuration. Note that in a robust log collection infrastructure, the relays have their own purpose, and Axoflow recommends running dedicated relay machines.
You can run several parallel relays to ensure horizontal redundancy. For example, if each of the relays has the same configuration, when one relay goes down another relay can take over the processing. Distribution of the logs can be done by the built-in client-side failover functionality and also by a general load balancer. The load balancer is also used to serve N+1 redundant relay deployments. In this case, switching from one relay to another relay is done when there is an outage but also for real load balancing purposes.
What AxoSyslog relays are not good for
The purpose of the relay is to buffer the logs for short term, for example, a few minutes or a few hours long outages (depending on the log volume). It is not designed to buffer logs generated by the sources during a very long server or connection outage, for example, up to a few days long.
If you expect extended outages, Axoflow recommends that you deploy servers instead of relays. There are many deployments where long term storage and archiving are performed on the central AxoSyslog server, but relays also do short-term log storage.
5.3.3 - Server mode
In server mode, AxoSyslog acts as a central log-collecting server. It receives messages from AxoSyslog clients and relays over the network, and stores them locally in files, or passes them to other applications, for example, log analyzers.
5.4 - Global objects
The AxoSyslog application uses the following objects:
Source driver: A communication method used to receive log messages. For example, AxoSyslog can receive messages from a remote host via TCP/IP, or read the messages of a local application from a file. For details on source drivers, see source: Read, receive, and collect log messages.
Source: A named collection of configured source drivers.
Destination driver: A communication method used to send log messages. For example, AxoSyslog can send messages to a remote host via TCP/IP, or write the messages into a file or database. For details on destination drivers, see destination: Forward, send, and store log messages.
Destination: A named collection of configured destination drivers.
Macro: An identifier that refers to a part of the log message. For example, the ${HOST} macro returns the name of the host that sent the message. Macros are often used in templates and filenames. For details, see Customize message format using macros and templates.
Rewrite rule: A rule modifies a part of the message, for example, replaces a string, or sets a field to a specified value. For details, see Modifying messages using rewrite rules.
Log paths: A combination of sources, destinations, and other objects like filters, parsers, and rewrite rules. The AxoSyslog application sends messages arriving from the sources of the log paths to the defined destinations, and performs filtering, parsing, and rewriting of the messages. Log paths are also called log statements. Log statements can include other (embedded) log statements and junctions to create complex log paths. For details, see log: Filter and route log messages using log paths, flags, and filters.
Template: A template is a set of macros that can be used to restructure log messages or automatically generate file names. For example, a template can add the hostname and the date to the beginning of every log message. For details, see Customize message format using macros and templates.
Option: Options set global parameters of syslog-ng, like the parameters of name resolution and timezone handling. For details, see Global options.
The AxoSyslog application receives the timezone and daylight saving information from the operating system it is installed on. If the operating system handles daylight saving correctly, so does syslog-ng.
The AxoSyslog application supports messages originating from different timezones. The original syslog protocol (RFC3164) does not include timezone information, but syslog-ng provides a solution by extending the syslog protocol to include the timezone in the log messages. The AxoSyslog application also enables administrators to supply timezone information for legacy devices which do not support the protocol extension.
5.5.1 - Assigning timezone to the message
When AxoSyslog receives a message, it assigns timezone information to the message using the following algorithm.
The sender application or host specifies the timezone of the messages. If the incoming message includes a timezone it is associated with the message. Otherwise, the local timezone is assumed.
Specify the time-zone() parameter for the source driver that reads the message. This timezone will be associated with the messages only if no timezone is specified within the message itself. Each source defaults to the value of the recv-time-zone() global option. It is not possible to override only the timezone information of the incoming message, but setting the keep-timestamp() option to no allows AxoSyslog to replace the full timestamp (timezone included) with the time the message was received.
Note
When processing a message that does not contain timezone information, the AxoSyslog application will use the timezone and daylight-saving that was effective when the timestamp was generated. For example, the current time is 2011-03-11 (March 11, 2011) in the EU/Budapest timezone. When daylight-saving is active (summertime), the offset is +02:00. When daylight-saving is inactive (wintertime) the timezone offset is +01:00. If the timestamp of an incoming message is 2011-01-01, the timezone associated with the message will be +01:00, but the timestamp will be converted, because 2011-01-01 meant winter time when daylight saving is not active but the current timezone is +02:00.
Specify the timezone in the destination driver using the time-zone() parameter. Each destination driver might have an associated timezone value: syslog-ng converts message timestamps to this timezone before sending the message to its destination (file or network socket). Each destination defaults to the value of the send-time-zone() global option.
Note
A message can be sent to multiple destination zones. The AxoSyslog application converts the timezone information properly for every individual destination zone.
Warning
If AxoSyslog sends the message is to the destination using the legacy-syslog protocol (RFC3164) which does not support timezone information in its timestamps, the timezone information cannot be encapsulated into the sent timestamp, so AxoSyslog will convert the hour:min values based on the explicitly specified timezone.
If the timezone is not specified, local timezone is used.
When macro expansions are used in the destination filenames, the local timezone is used. (Also, if the timestamp of the received message does not contain the year of the message, AxoSyslog uses the local year.)
Note
You can modify the timezone of the message using timezone-specific rewrite rules. For details, see Rewrite the timezone of a message.
5.5.2 - A note on timezones and timestamps
If the clients run syslog-ng, then use the ISO timestamp, because it includes timezone information. That way you do not need to adjust the recv-time-zone() parameter of syslog-ng.
If you want to output timestamps in Unix (POSIX) time format, use the S_UNIXTIME and R_UNIXTIME macros. You do not need to change any of the timezone related parameters, because the timestamp information of incoming messages is converted to Unix time internally, and Unix time is a timezone-independent time representation. (Actually, Unix time measures the number of seconds elapsed since midnight of Coordinated Universal Time (UTC) January 1, 1970, but does not count leap seconds.)
5.6 - Product licensing
Starting with version 3.2, the AxoSyslog application is licensed under a combined LGPL+GPL license. The core of AxoSyslog is licensed under the GNU Lesser General Public License Version 2.1 license, while the rest of the codebase is licensed under the GNU General Public License Version 2 license.
Note
Practically, the code stored under the lib directory of the source code package is under LGPL, the rest is GPL.
Multiple AxoSyslog servers can be run in fail-over mode. The AxoSyslog application does not include any internal support for this, as clustering support must be implemented on the operating system level. A tool that can be used to create UNIX clusters is Heartbeat (for details, see this page).
Starting with AxoSyslog version 3.2, AxoSyslog clients can be configured to send the log messages to failover servers in case the primary syslog server becomes unaccessible. For details on configuring failover servers, see the description of the failover-servers() destination option in destination: Forward, send, and store log messages.
5.8 - The structure of a log message
The following sections describe the structure of log messages. Currently there are two standard syslog message formats:
This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol. A syslog message consists of the following parts:
The total message cannot be longer than 1024 bytes.
The following is a sample syslog message:
<133>Feb 25 14:09:07 webserver syslogd: restart
The message corresponds to the following format:
<priority>timestamp hostname application: message
The different parts of the message are explained in the following sections.
Note
The AxoSyslog application supports longer messages as well. For details, see the log-msg-size() option in Global options reference. However, it is not recommended to enable messages larger than the packet size when using UDP destinations.
The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while Severity marks its importance.
PRI formula
The Priority value is calculated using the following formula:
<PRI> =( <facility> * 8) + <severity>
That is, you first multiply the Facility number by 8, and then add the numerical value of the Severity to the multiplied sum.
Example: the correlation between facility value, severity value, and the Priority value in the PRI message part
The following example illustrates a sample syslog message with a sample PRI field (that is, Priority value):
<133> Feb 25 14:09:07 webserver syslogd: restart
In this example, <133> represents the PRI field (Priority value). The syslog message’s Facility value is 16, and the Severity value is 5.
Substituting the numerical values into the <PRI> = ( <facility> * 8) + <severity> formula, the results match the Priority value in our example:
The HEADER message part contains a timestamp and the hostname (without the domain name) or the IP address of the device. The timestamp field is the local time in the Mmm dd hh:mm:ss format, where:
Mmm is the English abbreviation of the month: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.
dd is the day of the month on two digits. If the day of the month is less than 10, the first digit is replaced with a space. (for example, Aug 7.)
hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between 00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive.
Note
The AxoSyslog application supports other timestamp formats as well, like ISO, or the PIX extended format. For details, see the ts-format() option in Global options reference.
The MSG part contains the name of the program or process that generated the message, and the text of the message itself. The MSG part is usually in the following format: program[pid]: message text.
5.8.2 - IETF-syslog messages
This section describes the format of a syslog message, according to the IETF-syslog protocol. A syslog message consists of the following parts:
The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second.
The message originated from a host that identifies itself as “mymachine.example.com”.
The APP-NAME is “su” and the PROCID is unknown.
The MSGID is “ID47”.
The MSG is “‘su root’ failed for lonvick…”, encoded in UTF-8.
In this example, the encoding is defined by the BOM:
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
There is no STRUCTURED-DATA present in the message, this is indicated by “-” in the STRUCTURED-DATA field.
The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.
The PRI message part
The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.
Note
Facility codes may slightly vary between different platforms. The AxoSyslog application accepts facility codes as numerical values as well.
syslog Message Facilities
Numerical Code
Facility
0
kernel messages
1
user-level messages
2
mail system
3
system daemons
4
security/authorization messages
5
messages generated internally by syslogd
6
line printer subsystem
7
network news subsystem
8
UUCP subsystem
9
clock daemon
10
security/authorization messages
11
FTP daemon
12
NTP subsystem
13
log audit
14
log alert
15
clock daemon
16-23
locally used facilities (local0-local7)
The following table lists the severity values.
Numerical Code
Severity
0
Emergency: system is unusable
1
Alert: action must be taken immediately
2
Critical: critical conditions
3
Error: error conditions
4
Warning: warning conditions
5
Notice: normal but significant condition
6
Informational: informational messages
7
Debug: debug-level messages
syslog Message Severities
The HEADER message part
The HEADER part contains the following elements:
VERSION: Version number of the syslog protocol standard. Currently this can only be 1.
ISOTIMESTAMP: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00.
HOSTNAME: The machine that originally sent the message.
APPLICATION: The device or application that generated the message
PID: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.
MESSAGEID: The ID number of the message.
Note
The AxoSyslog application supports other timestamp formats as well, like ISO, or the PIX extended format. The timestamp used in the IETF-syslog protocol is derived from RFC3339, which is based on ISO8601. For details, see the ts-format() option in Global options reference.
The AxoSyslog application will truncate the following fields:
If APP-NAME is longer than 48 characters it will be truncated to 48 characters.
If PROC-ID is longer than 128 characters it will be truncated to 128 characters.
If MSGID is longer than 32 characters it will be truncated to 32 characters.
If HOSTNAME is longer than 255 characters it will be truncated to 255 characters.
The STRUCTURED-DATA message part
The STRUCTURED-DATA message part may contain meta- information about the syslog message, or application-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists of data blocks enclosed in brackets ([]). Every block includes the ID of the block, and one or more name=value pairs. The AxoSyslog application automatically parses the STRUCTURED-DATA part of syslog messages, which can be referenced in macros (for details, see Macros of AxoSyslog). An example STRUCTURED-DATA block looks like:
The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if the BOM
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
character is present in the message. If the message does not contain the BOM character, the encoding is treated as unknown. Usually messages arriving from legacy sources do not include the BOM character. CRLF characters will not be removed from the message.
5.8.3 - Enterprise-wide message model (EWMM)
The following section describes the structure of log messages using the Enterprise-wide message model or EWMM message format.
The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving AxoSyslog component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first AxoSyslog component in your infrastructure has received it, which is important if you want to forward a message for example, to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.
The following is a sample log message in EWMM format.
<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
The message has the following parts:
The header of the complies with the RFC5424 message format, where the PROGRAM field is set to @syslog-ng, and the SDATA field is empty.
The MESSAGE part is in JSON format, and contains the actual message, as well as any name-value pairs that AxoSyslog has attached to or extracted from the message. The ${._TAGS} field contains the identifier of the AxoSyslog source that has originally received the message on the first AxoSyslog node.
When the AxoSyslog application receives a message, it automatically parses the message. The AxoSyslog application can automatically parse log messages that conform to the RFC3164 (BSD or legacy-syslog) or the RFC5424 (IETF-syslog) message formats. If AxoSyslog cannot parse a message, it results in an error.
Note
In case you need to relay messages that cannot be parsed without any modifications or changes, use the flags(no-parse) option in the source definition, and a template containing only the ${MESSAGE} macro in the destination definition.
To parse non-syslog messages, for example, JSON, CSV, or other messages, you can use the built-in parsers of AxoSyslog. For details, see parser: Parse and segment structured messages.
A parsed syslog message has the following parts:
Timestamps
Two timestamps are associated with every message: one is the timestamp contained within the message (that is, when the sender sent the message), the other is the time when AxoSyslog has actually received the message.
Severity
The severity of the message.
Facility
The facility that sent the message.
Tags
Custom text labels added to the message that are mainly used for filtering. None of the current message transport protocols adds tags to the log messages. Tags can be added to the log message only within AxoSyslog. The AxoSyslog application automatically adds the id of the source as a tag to the incoming messages. Other tags can be added to the message by the pattern database, or using the tags() option of the source.
IP address of the sender
The IP address of the host that sent the message. Note that the IP address of the sender is a hard macro and cannot be modified within AxoSyslog but the associated hostname can be modified, for example, using rewrite rules.
Hard macros
Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. The most important consideration with hard macros is that they are read-only, meaning they cannot be modified using rewrite rules or other means.
Soft macros
Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the AxoSyslog pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within AxoSyslog, for example, using rewrite rules.
Note
It is also possible to set the value of built-in soft macros using parsers, for example, to set the ${HOST} macro from the message using a column of a CSV-parser.
The data extracted from the log messages using named pattern parsers in the pattern database are also soft macros.
Internally, AxoSyslog represents every message as UTF-8. The maximal length of the log messages is limited by the log-msg-size() option: if a message is longer than this value, AxoSyslog truncates the message at the location it reaches the log-msg-size() value, and discards the rest of the message.
When encoding is set in a source (using the encoding() option) and the message is longer (in bytes) than log-msg-size() in UTF-8 representation, AxoSyslog splits the message at an undefined location (because the conversion between different encodings is not trivial).
5.10 - Structuring macros, metadata, and other value-pairs
Available in AxoSyslog 3.3 and later.
The AxoSyslog application allows you to select and construct name-value pairs from any information already available about the log message, or extracted from the message itself. You can directly use this structured information, for example, in the following places:
Prior to version 4.0, AxoSyslog handled every data as strings, and allowed you to convert the strings into other types of data that only certain destinations data formats supported. For example, SQL, MongoDB, JSON, or AMQP support data types like numbers or dates. The AxoSyslog application allows you to specify the data type in templates (this is also called type-hinting or type-casting). If the destination driver supports data types, AxoSyslog converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.
Starting with AxoSyslog 4.0, each name-value pair is a (name, type, value) triplet, and several components of AxoSyslog have typing support, for example, json-parser() and the $(format-json) template function. For details, see the list of supported data types.
Using explicit type-hinting
You can explicitly type-cast a AxoSyslog template to a specific type. To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID"). See the Type-hinting examples and the list of supported data types for details.
Warning
Hazard of data loss! If AxoSyslog cannot convert the data into the specified type, an error occurs, and AxoSyslog drops the message by default. To change how AxoSyslog handles data-conversion errors, see Global options.
The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.
The following example formats the MESSAGE field as a JSON list.
$(format-json message=list($MESSAGE))"
Data types in AxoSyslog
The AxoSyslog application currently supports the following data-types.
boolean: Converts the data to a boolean value. Anything that begins with a t or 1 is converted to true, anything that begins with an f or 0 is converted to false.
datetime: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the $UNIXTIME macro for this purpose.
double: A floating-point number.
json: A JSON snippet. (Available in AxoSyslog 4.0 and later.)
integer: A 32-bit or 64-bit integer, determined by the destination. For example, mongodb uses int32 if the number is less than MAXINT32 and int64 otherwise.
string: The data as a string.
Components that support data types
In AxoSyslog 4.0 and later, the following AxoSyslog components that support data types. Other components treat every data as strings.
Comparisons in filter expressions: the previously numeric operators are type-aware. The exact comparison depends on the types associated with the values you compare. For details, see Comparing macro values in filters.
When using the json-parser(), AxoSyslog converts all elements of the JSON object to name-value pairs. Any type information originally present in the incoming JSON object is retained, and automatically propagated to other AxoSyslog components (for example, a destination) if they support types.
Elements without a type are treated as strings.
JSON lists (arrays) are converted to AxoSyslog lists, so you can manipulate them using the $(list-*) template functions.
You can set the type of the field. Where you can use of templates in set() and groupset(), you can use type-casting, and the type information is properly promoted. For details, see Specifying data types in value-pairs.
db-parser(): The db-parser() rules can associate types with values using the "type" attribute, for example:
<valuename="foobar"type="integer">$PID</value>
The integer is a type-cast that associates $foobar with an integer type. db-parser()’s internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair.
add-contextual-data(): Name-value pairs that are populated using add-contextual-data() propagate type information, similarly to db-parser().
SQL type support: The sql() driver supports types, so that columns with specific types are stored as those types.
Template type support: You can cast templates explicitly to a specific type. Templates also propagate type information from macros, template functions, and values in the template string.
python() typing: All Python components (sources, destinations, parsers, and template functions) support all data types, except json().
On-disk serialized formats (that is, disk buffer): Version 4.0 and newer are compatible with messages serialized with an earlier version, and the format is compatible for downgrades as well. This means that even if a newer version of AxoSyslog serialized a message, older versions and associated tools are able to read it (but drop the type information of course).
5.10.2 - value-pairs()
Type:
parameter list of the value-pairs() option
Default:
empty string
Description: The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.
Example: Using the value-pairs() option
The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message.
Every macro is included in the selection only once, but redundant information may appear if multiple macros include the same information (for example, including several date-related macros in the selection).
value-pairs() parameters
The value-pairs() option has the following parameters. The parameters are evaluated in the following order:
Space-separated list of macros to remove from the selection created using the scope() option.
Default:
empty string
Description: This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the scope() parameter. For example, the following example removes the SDATA macros from the selection.
The name of the macro to remove can include wildcards (*, ?). Regular expressions are not supported.
key()
Type:
Space-separated list of macros to be included in the selection.
Default:
empty string
Description: This option selects the specified macros. The selected macros will be included as MACRONAME = MACROVALUE, that is using key("HOST") will result in HOST = $HOST. You can use wildcards (*, ?) to select multiple macros. For example:
Description: If this option is specified, AxoSyslog does not include value-pairs with empty values in the output. For example: $(format-json --scope none --omit-empty-values) or
Description: This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:
<pattern-to-select-names>, <list of transformations>
Default:
empty string
Description: This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the rekey() option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see glob.
Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, replace-prefix(".class", ".patterndb") changes the beginning tag .class to .patterndb
This option was called replace() in AxoSyslog version 3.4.
lower
Convert all keys to lowercase. Only supports US ASCII.
shift("<number>")
Cuts the specified number of characters from the beginning of the name.
shift-levels("<number>")
Similar to –shift, but instead of cutting characters, it cuts dot-delimited “levels” in the name (including the initial dot). For example, --shift-levels 2 deletes the prefix up to the second dot in the name of the key: .iptables.SRC becomes SRC
upper
Convert all keys to uppercase. Only supports US ASCII.
Example: Using the rekey() option
The following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters from the names, and adds a new prefix (events.).
Space-separated list of macro groups to be included in the selection.
Default:
empty string
Description: This option selects predefined groups of macros. The following groups are available:
nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within AxoSyslog and are not originally part of the message, therefore are not included in this group.
dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with a dot character are generated within AxoSyslog and are not originally part of the message.
all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs and dot-nv-pairs.
rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $DATE.
rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, $R_DATE, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA..
The rfc5424 group also has the following alias: syslog-proto. Note that the value of $R_DATE will be listed under the DATE key.
The rfc5424 group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the $SOURCEIP macro), use the selected-macros group instead.
all-macros: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).
selected-macros: Include the macros of the rfc3164 groups, and the most commonly used metadata about the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.
sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA.
everything: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).
none: Reset previously added scopes, for example, to delete automatically-added name-value pairs. The following example deletes every value-pair from the scope, and adds only the ones starting with iptables: $(format-welf --scope none .iptables.*)
For example:
value-pairs( scope(rfc3164 selected-macros))
5.11 - Things to consider when forwarding messages between AxoSyslog hosts
When you send your log messages from a AxoSyslog client through the network to a AxoSyslog server, you can use different protocols and options. Every combination has its advantages and disadvantages. The most important thing is to use matching protocols and options, so the server handles the incoming log messages properly.
In AxoSyslog you can change many aspects of the network communication. First of all, there is the structure of the messages itself. Currently, AxoSyslog supports two standard syslog protocols: the BSD (RFC3164) and the syslog (RFC5424) message format.
These RFCs describe the format and the structure of the log message, and add a (lightweight) framing around the messages. You can set this framing/structure by selecting the appropriate driver in AxoSyslog. There are two drivers you can use: the network() driver and the syslog() driver. The syslog() driver is for the syslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.
The tcp() and udp() drivers are now deprecated, they are essentially equivalent with the network(transport(tcp)) and network(transport(udp)) drivers.
In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCP and UDP, and optionally also higher-level transport protocols: TLS (over TCP. To complicate things a bit more, you can configure the network() driver (corresponding to the BSD (RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used in RFC5424) using the flag(syslog-protocol) option.
Because some combination of drivers and options are invalid, you can use the following drivers and options as sources and as destinations:
If you use the same driver and options in the destination of your AxoSyslog client and the source of your AxoSyslog server, everything should work as expected. Unfortunately there are some other combinations, that seem to work, but result in losing parts of the messages. The following table show the combinations:
Source \ Destination
syslog/tcp
syslog/udp
syslog/tls
network/tcp
network/udp
network/tls
network/tcp/flag
network/udp/flag
network/tls/flag
syslog/tcp
✔
-
-
!
-
-
!
-
-
syslog/udp
-
✔
-
-
!
-
-
!
-
syslog/tls
-
-
✔
-
-
!
-
-
!
network/tcp
-
-
-
✔
-
-
✔?
-
-
network/udp
-
✔?
-
-
✔
-
-
✔?
-
network/tls
-
-
-
-
-
✔
-
-
✔?
network/tcp/flag
!
-
-
!
-
-
✔
-
-
network/udp/flag
-
!
-
-
!
-
-
✔
-
network/tls/flag
-
-
!
-
-
!
-
-
✔
Source-destination driver combinations
- This method does not work. The logs will not get to the server.
✔ This method works.
! This method has some visible drawbacks. The logs go through, but some of the values are missing/misplaced/and so on.
✔? This method seems to work, but it is not recommended because this can change in a future release.
6 - The configuration file
6.1 - Location of the configuration file
To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.
Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog collector Helm chart.
6.2 - The configuration syntax in detail
Every syslog-ng.conf configuration file must begin with a line containing version information. For syslog-ng version 4.5, this line looks like:
@version: 4.5.0
If the configuration file does not contain the version information, syslog-ng assumes that the file is for version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later can operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.
@version: current sets the configuration version to the currently installed version.
Example: A simple configuration file
The following is a very simple configuration file for syslog-ng: it collects the internal messages of syslog-ng and the messages from /dev/log into the /var/log/messages_syslog-ng.log file.
The syslog-ng’s config file format was written by programmers for programmers to be understood by programmers. That may not have been the stated intent, but it is how things turned out. The syntax is exactly that of C, all the way down to braces and statement terminators.
The main body of the configuration file consists of object definitions: sources, destinations, logpaths define which log message are received and where they are sent. All identifiers, option names and attributes, and any other strings used in the syslog-ng.conf configuration file are case sensitive. Object definitions (also called statements) have the following syntax:
Type of the object: One of source, destination, log, filter, parser, rewrite rule, or template.
Identifier of the object: A unique name identifying the object. When using a reserved word as an identifier, enclose the identifier in quotation marks.
All identifiers, attributes, and any other strings used in the syslog-ng.conf configuration file are case sensitive.
Note
<p>Use identifiers that refer to the type of the object they identify. For example, prefix source objects with <code>s_</code>, destinations with <code>d_</code>, and so on.</p>
Repeating a definition of an object (that is, defining the same object with the same id more than once) is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file.
Parameters: The parameters of the object, enclosed in braces {parameters}.
Semicolon: Object definitions end with a semicolon (;).
For example, the following line defines a source and calls it s_internal.
source s_internal { internal();};
The object can be later referenced in other statements using its ID, for example, the previous source is used as a parameter of the following log statement:
log { source(s_internal); destination(d_file);};
The parameters and options within a statement are similar to function calls of the C programming language: the name of the option followed by a list of its parameters enclosed within brackets and terminated with a semicolon.
For example, the file() driver in the following source statement has three options: the filename (/var/log/apache/access.log), follow-freq(), and flags(). The follow-freq() option also has a parameter, while the flags() option has two parameters.
Objects may have required and optional parameters. Required parameters are positional, meaning that they must be specified in a defined order. Optional parameters can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object.
Example: Using required and optional parameters
The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect:
Some options are global options, or can be set globally, for example, whether AxoSyslog should use DNS resolution to resolve IP addresses. Global options are detailed in Global options.
options { use-dns(no);};
Objects can be used before definition.
Objects can be defined inline as well. This is useful if you use the object only once (for example, a filter). For details, see Defining configuration objects inline.
To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.
# Comment: This is a stream sourcesource s_demo_stream { unix-stream("<path-to-socket>" max-connections(10) group(log));};
Note
Before activating a new configuration, check that your configuration file is syntactically correct using the syslog-ng --syntax-only command.
To activate the configuration, reload the configuration using the /etc/init.d/syslog-ng reload command.
6.3 - Notes about the configuration syntax
When you are editing the syslog-ng.conf configuration file, note the following points:
The configuration file can contain a maximum of 6665 source / destination / log elements.
When writing the names of options and parameters (or other reserved words), the hyphen (-) and underscore (_) characters are equivalent, for example, max-connections(10) and max_connections(10) are both correct.
Numbers can be prefixed with + or - to indicate positive or negative values. Numbers beginning with zero (0) or 0x are treated as octal or hexadecimal numbers, respectively.
Starting with AxoSyslog version 3.5, you can use suffixes for kilo-, mega-, and gigabytes. Use the Kb, Mb, or Gb suffixes for the base-10 version, and Kib, Mib, or Gib for the base-2 version. That is, 2MB means 2000000, while 2MiB means 2097152. For example, to set the log-msg-size() option to 2000000 bytes, use log-msg-size(2Mb).
You can use commas (,) to separate options or other parameters for readability, AxoSyslog completely ignores them. The following declarations are equivalent:
When enclosing object IDs (for example, the name of a destination) between double-quotes ("mydestination"), the ID can include whitespace as well, for example:
Starting with AxoSyslog 3.4, you can define configuration objects inline, where they are actually used, without having to define them in a separate placement. This is useful if you need an object only once, for example, a filter or a rewrite rule. Every object can be defined inline: sources, destinations, filters, parsers, rewrite rules, and so on.
To define an object inline, use braces instead of parentheses. That is, instead of <object-type> (<object-id>);, you use <object-type> {<object-definition>};
Example: Using inline definitions
The following two configuration examples are equivalent. The first one uses traditional statements, while the second uses inline definitions.
Starting with AxoSyslog 3.4, every configuration object is a log expression. Every configuration object is essentially a configuration block, and can include multiple objects. To reference the block, only the top-level object must be referenced. That way you can use embedded log statements, junctions and in-line object definitions within source, destination, filter, rewrite and parser definitions. For example, a source can include a rewrite rule to modify the messages received by the source, and that combination can be used as a simple source in a log statement. This feature allows you to preprocess the log messages very close to the source itself.
To embed multiple objects into a configuration object, use the following syntax. Note that you must enclose the configuration block between braces instead of parenthesis.
For example, to process a log file in a specific way, you can define the required processing rules (parsers and rewrite expressions) and combine them in a single object:
The s_apache source uses a file source (the error log of an Apache webserver) and references a specific parser to process the messages of the error log. The log statement references only the s_apache source, and any other object in the log statement can already use the results of the p_apache_parserparser.
Note
You must start the object definition with a channel even if you will use a junction, for example:
If you want to embed configuration objects into sources or destinations, always use channels, otherwise the source or destination will not behave as expected. For example, the following configuration is good:
You can define global variables in the configuration file. Global variables are actually name-value pairs. When syslog-ng processes the configuration file during startup, it automatically replaces name with value. To define a global variable, use the following syntax:
@define name "value"
The value can be any string, but special characters must be escaped (for details, see Regular expressions). To use the variable, insert the name of the variable enclosed between backticks (`, similarly to using variables in Linux or UNIX shells) anywhere in the configuration file. If backticks are meant literally, repeat the backticks to escape them. For example:
``not-substituted-value``
The value of the global variable can be also specified using the following methods:
Without any quotes, as long as the value does not contain any spaces or special characters. In other words, it contains only the following characters: a-zA-Z0-9_..
Between apostrophes, in case the value does not contain apostrophes.
Between double quotes, in which case special characters must be escaped using backslashes (\\).
Note
The environmental variables of the host are automatically imported and can be used as global variables.
In AxoSyslog 3.24 and later, the location of the configuration file is available as the syslog-ng-sysconfdir variable.
Example: Using global variables
For example, if an application is creating multiple log files in a directory, you can store the path in a global variable, and use it in your source definitions.
Starting with AxoSyslog version 4.2, you can specify a configuration identifier in the syslog-ng.conf file, for example:
@config-id: cfg-20230404-13-g02b0850fc
This can be useful in managed environments, where AxoSyslog instances and their configuration are automatically deployed or generated.
To show the configuration ID, run syslog-ng-ctl config --id
This returns the ID of the currently active configuration, and the SHA256 hash of the configuration (the hash of the output of the syslog-ng-ctl config --preprocessed command). The output is similar to:
To increase its flexibility and simplify the development of additional modules, the AxoSyslog application is modular. The majority of AxoSyslog’s functionality is in separate modules. As a result, it is also possible to fine-tune the resource requirements of AxoSyslog (for example, by loading only the modules that are actually used in the configuration, or simply omitting modules that are not used but require large amount of memory).
Each module contains one or more plugins that add some functionality to AxoSyslog (for example, a destination or a source driver).
To display the list of available modules, run the syslog-ng --version command.
To display the description of the available modules, run the syslog-ng --module-registry command.
To customize which modules AxoSyslog automatically loads when AxoSyslog starts, use the --default-modules command-line option of AxoSyslog.
To request loading a module from the AxoSyslog configuration file, see Loading modules.
For details on the command-line parameters of AxoSyslog mentioned in the previous list, see the AxoSyslog man page at The syslog-ng manual page.
6.8.1 - Loading modules
The AxoSyslog application loads every available module during startup.
To load a module that is not loaded automatically, include the following statement in the AxoSyslog configuration file:
@module <module-name>
Note the following points about the @module statement:
The @module statement is a top-level statement, that is, it cannot be nested into any other statement. It is usually used immediately after the @version statement.
Every @module statement loads a single module: loading multiple modules requires a separate @module statement for every module.
In the configuration file, the @module statement of a module must be earlier than the module is used.
Note
To disable loading every module automatically, set the autoload-compiled-modules global variable to 0 in your configuration file:
@define autoload-compiled-modules 0
Note that in this case you have to explicitly load the modules you want to use.
Use the @requires statement to ensure that the specified module is loaded
To ensure that a module is loaded, include the following statement in the AxoSyslog configuration file or the external files included in the configuration file:
@requires <module-name>
Note
If you include the @requires statement in the:
AxoSyslog configuration file, AxoSyslog attempts to load the required module. If it fails to load the module, AxoSyslog stops and an error message is displayed.
external files included in the configuration file, AxoSyslog attempts to load the required module. If it fails to load the module, only the external file is not processed.
Note that this is not true for modules marked as mandatory. You can make a dependency module mandatory by defining an error message after the @requires <module-name> statement, for example:
Example
@requires http "The http() driver is required for elasticsearch-http(). Install syslog-ng-mod-http to continue."
6.8.2 - Listing configuration options
Starting with AxoSyslog 3.25, you can use the syslog-ng-cfg-db.py utility to list the available options of configuration objects. For example, you can list all the options that can be set in the file source, and so on.
The syslog-ng-cfg-db.py utility has the following options:
The following command lists the contexts that the utility supports.
syslog-ng-cfg-db.py
Note
Currently, sources and destinations are supported.
The following command lists the available drivers of a context:
syslog-ng-cfg-db.py -c <source|destination>
The following command lists the available options of a specific driver and specifies the context and the driver:
The script caches the list of the options, so if you want to rebuild the database, you have to use the -r option.
6.8.3 - Visualize the configuration
Starting with AxoSyslog 3.25, you can visualize the configuration of a running AxoSyslog instance using the syslog-ng-ctl --export-config-graph command. The command walks through the effective configuration, and exports it as a graph into a JSON structure.
The resulting JSON file can be converted into DOT file format that visualization tools (for example, Graphviz) can use. The package includes a Python script to convert the exported JSON file into DOT format: <syslog-ng-installation-directory>/contrib/scripts/config-graph-json-to-dot.py
You can convert the DOT file into PNG or PDF format using external tools.
6.9 - Managing complex configurations
The following sections describe some methods that can be useful to simplify the management of large-scale AxoSyslog installations.
6.9.1 - Including configuration files
The AxoSyslog application supports including external files in its configuration file, so parts of its configuration can be managed separately. To include the contents of a file in the AxoSyslog configuration, use the following syntax:
@include "<filename>"
This imports the entire file into the configuration of AxoSyslog, at the location of the include statement. The <filename> can be one of the following:
A filename, optionally with full path. The filename (not the path) can include UNIX-style wildcard characters (*, ?). When using wildcard characters, AxoSyslog will include every matching file. For details on using wildcard characters, see Types and options of regular expressions.
A directory. When including a directory, AxoSyslog will try to include every file from the directory, except files beginning with a ~ (tilde) or a . (dot) character. Including a directory is not recursive. The files are included in alphabetic order, first files beginning with uppercase characters, then files beginning with lowercase characters. For example, if the directory contains the a.conf, B. conf, c.conf, D.conf files, they will be included in the following order: B.conf, D. conf, a.conf, c.conf.
When including configuration files, consider the following points:
The default path where AxoSyslog looks for the file depends on where AxoSyslog is installed. The syslog-ng --version command displays this path as Include-Path.
Defining an object twice is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file. If an object is defined twice (for example, the original configuration file and the file imported into this configuration file both define the same option, source, or other object), then the object that is defined later in the configuration file will be effective. For example, if you set a global option at the beginning of the configuration file, and later include a file that defines the same option with a different value, then the option defined in the imported file will be used.
Files can be embedded into each other: the included files can contain include statements as well, up to a maximum depth of 15 levels.
You cannot include complete configuration files into each other, only configuration snippets can be included. This means that the included file cannot have a @version statement.
Include statements can only be used at top level of the configuration file. For example, the following is correct:
@version: 4.5.0
@include "example.conf"
But the following is not:
source s_example { @include "example.conf"};
Warning
The AxoSyslog application will not start if it cannot find a file that is to be included in its configuration. Always double-check the filenames, paths, and access rights when including configuration files, and use the --syntax-only command-line option to check your configuration.
6.9.2 - Reusing configuration blocks
To create a reusable configuration snippet and reuse parts of a configuration file, you have to define the block (for example, a source) once, and reference it later. Any AxoSyslog object can be a block. Use the following syntax to define a block:
block type name(){<contents of the block>};
Type must be one of the following: destination, filter, log, options, parser, rewrite, root, source. The root blocks can be used in the “root” context of the configuration file, that is, outside any other statements.
Note that options can be used in blocks only in version 3.22 and later.
Blocks may be nested into each other, so for example, a block can be built from other blocks. Blocks are somewhat similar to C++ templates.
The type and name combination of each block must be unique, that is, two blocks can have the same name if their type is different.
To use a block in your configuration file, you have to do two things:
Include the file defining the block in the syslog-ng.conf file — or a file already included into syslog-ng.conf. Version 3.7 and newer automatically includes the *.conf files from the <directory-where-syslog-ng-is-installed>/scl/*/ directories.
Reference the name of the block in your configuration file. This will insert the block into your configuration. For example, to use a block called myblock, include the following line in your configuration:
myblock()
Blocks may have parameters, but even if they do not, the reference must include opening and closing parentheses like in the previous example.
The contents of the block will be inserted into the configuration when AxoSyslog is started or reloaded.
Example: Reusing configuration blocks
Suppose you are running an application on your hosts that logs into the /opt/var/myapplication.log file. Create a file (for example, myblocks.conf) that stores a source describing this file and how it should be read:
To define a block that defines more than one object, use root as the type of the block, and reference the block from the main part of the AxoSyslog configuration file.
Example: Defining blocks with multiple elements
The following example defines a source, a destination, and a log path to connect them.
Since the block is inserted into the AxoSyslog configuration when AxoSyslog is started, the block can be generated dynamically using an external script if needed. This is useful when you are running AxoSyslog on different hosts and you want to keep the main configuration identical.
If you want to reuse more than a single configuration object, for example, a logpath and the definitions of its sources and destinations, use the include feature to reuse the entire snippet. For details, see Including configuration files.
Mandatory parameters
You can express in block definitons that a parameter is mandatory by defining it with empty brackets (). In this case, the parameter must be overridden in the reference block. Failing to do so will result in an error message and initialization failure.
To make a parameter expand into nothing (for example, because it has no default value, like hook-commands() or tls()), insert a pair of double quote marks inside the empty brackets: ("")
Example: Mandatory parameters
The following example defines a TCP source that can receive the following parameters: the port where AxoSyslog listens (localport), and optionally source flags (flags).
Because localport is defined with empty brackets (), it is a mandatory parameter. However, the flags parameter is not mandatory, because it is defined with an empty double quote bracket pair (""). If you do not enter a specific value when referencing this parameter, the value will be an empty string. This means that in this case
my_tcp_source(localport(8080))
will be expanded to:
network(port(8080) transport(tcp) flags());
Passing arguments to configuration blocks
Configuration blocks can receive arguments as well. The parameters the block can receive must be specified when the block is defined, using the following syntax:
block type block_name(argument1(<default-value-of-the-argument>) argument2(<default-value-of-the-argument>) argument3())
If an argument does not have a default value, use an empty double quote bracket pair ("") after the name of the argument. To refer the value of the argument in the block, use the name of the argument between backticks, for example:
`argument1`
Example: Passing arguments to blocks
The following sample defines a file source block, which can receive the name of the file as a parameter. If no parameter is set, it reads messages from the /var/log/messages file.
If you reference the block with more arguments then specified in its definition, you can use these additional arguments as a single argument-list within the block. That way, you can use a variable number of optional arguments in your block. This can be useful when passing arguments to a template, or optional arguments to an underlying driver.
The three dots (…) at the end of the argument list refer to any additional parameters. It tells AxoSyslog that this macro accepts __VARARGS__, therefore any name-value pair can be passed without validation. To reference this argument-list, insert __VARARGS__ to the place in the block where you want to insert the argument-list. Note that you can use this only once in a block.
The following definition extends the logfile block from the previous example, and passes the optional arguments (follow-freq(1) flags(no-parse)) to the file() source.
6.9.3 - Generating configuration blocks from a script
Purpose:
The AxoSyslog application can automatically execute scripts when it is started, and can include the output of such script in the configuration file. To create and use a script that generates a part of the AxoSyslog configuration file (actually, a configuration block), complete the following steps. The steps include examples for collecting Apache access log files (access.log) from subdirectories, but you can create any script that creates a valid AxoSyslog configuration snippet.
Steps:
Navigate to the directory where you have installed AxoSyslog (for example, /opt/syslog-ng/share/include/scl/), and create a new directory, for example, apache-access-logs. The name of the directory will be used in the AxoSyslog configuration file as well, so use a descriptive name.
Create a file called plugin.conf in this new directory.
Edit the plugin.conf file and add the following line:
Replace <directory-name> with the name of the directory (for example, apache-access-logs), and <my-script> with the filename of your script (for example, apache-access-logs.sh). You can reference the script in your AxoSyslog configuration file as a configuration block using the value name option.
The context option determines the type of the configuration snippet that the script generates, and must be one of the following: destination, filter, log, parser, rewrite, root, source. The root blocks can be used in the “root” context of the configuration file, that is, outside any other statements. In the example, context(source) means that the output of the script will be used within a source statement.
You can pass parameters to the script. In the script these parameters are available as environment variables, and have the confgen_ prefix. For example, passing the --myparameter parameter becomes available in the script as the confgen_myparameter environment variable.
Write a script that generates the output you need, and formats it to a configuration snippet that AxoSyslog can use. The filename of the script must match with the filename used in plugin.conf, for example, apache-access-logs.sh.
The following example checks the /var/log/apache2/ directory and its subdirectories, and creates a source driver for every directory that contains an access.log file.
#!/bin/bashfor i in `find /var/log/apache2/ -type d`;doecho"file(\"$i/access.log\" flags(no-parse) program-override(\"apache2\"));";done;
The script generates an output similar to this one, where service* is the actual name of a subdirectory:
Include the plugin.conf file in the syslog-ng.conf file — or a file already included into syslog-ng.conf. Version 3.7 and newer automatically includes the *.conf files from the <directory-where-syslog-ng-is-installed>/scl/*/ directories. For details on including configuration files, see Including configuration files.
Add the block you defined in the plugin.conf file to your AxoSyslog configuration file. You can reference the block using the value of the name option from the plugin.conf file, followed by parentheses, for example, apache-access-logs(). Make sure to use the block in the appropriate context of the configuration file, for example, within a source statement if the value of the context option in the plugin.conf file is source.
Instead of writing Python code into your AxoSyslog configuration file, you can store the Python code for your Python object in an external file. That way, it is easier to write, maintain, and debug the code. You can store the Python code in any directory in your system, but make sure to include it in your Python path.
When referencing a Python class from an external file in the class() option of a Python block in the AxoSyslog configuration file, the class name must include the name of the Python file containing the class, without the path and the .py extension. For example, if the MyDestination class is available in the /etc/syslog-ng/etc/pythonexample.py file, use class("pythonexample.MyDestination"):
Starting with 3.26, AxoSyslog assigns a persist name to Python sources and destinations. The persist name is generated from the class name. If you want to use the same Python class multiple times in your AxoSyslog configuration, add a unique persist-name() to each source or destination, otherwise AxoSyslog will not start. For example:
Alternatively, you can include the following line in the Python package: @staticmethod generate_persist_name. For example:
from syslogng import LogSource
class PyNetworSource(LogSource):
@staticmethod
def generate_persist_name(options):
return options["port"] def run(self):
pass
def request_exit(self):
pass
If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.
In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.
To debug and troubleshoot your Python code, AxoSyslog allows you to use the logger() method to send log messages to the internal() source of AxoSyslog. That way the diagnostic messages of your Python code are treated the same way as other such log messages of AxoSyslog. This has the following benefits:
The logger() method respects the log level settings of AxoSyslog. You can write error, warning, info, debug, and trace level messages.
You can follow what your Python code is doing even if AxoSyslog is running as a daemon in the background.
Logging to the internal() source is available in AxoSyslog version 3.20 and later.
To send log messages to the internal() source from Python
Add the following import to your Python code:
import syslogng
Create a logger object:
logger= syslogng.Logger()
Use the logger object in your Python code, for example:
logger.info("This is a sample log message send from the Python code.")
You can use the following log levels: logger.error, logger.warning, logger.info, logger.debug, logger.trace
Make sure that your AxoSyslog configuration includes the internal() source, for example:
If the message received by the source does not have a proper syslog header, you can use the default-facility() and default-priority() options to set the facility and priority of the messages. Note that these values are applied only to messages that do not set these parameters in their header.
Define a source only once. The same source can be used in several log paths. Duplicating sources causes AxoSyslog to open the source (TCP/IP port, file, and so on) more than once, which might cause problems. For example, include the /dev/log file source only in one source statement, and use this statement in more than one log path if needed.
Warning
Sources and destinations are initialized only when they are used in a log statement. For example, AxoSyslog starts listening on a port or starts polling a file only if the source is used in a log statement. For details on creating log statements, see log: Filter and route log messages using log paths, flags, and filters.
To collect log messages on a specific platform, it is important to know how the native syslogd communicates on that platform. The following table summarizes the operation methods of syslogd on some of the tested platforms:
Communication methods used between the applications and syslogd
Platform
Method
Linux
A SOCK_DGRAM unix socket named /dev/log. Newer distributions that use systemd collect log messages into a journal file.
BSD flavors
A SOCK_DGRAM unix socket named /var/run/log.
Solaris (2.5 or below)
An SVR4 style STREAMS device named /dev/log.
Solaris (2.6 or above)
In addition to the STREAMS device used in earlier versions, 2.6 uses a new multithreaded IPC method called door. By default the door used by syslogd is /etc/.syslog_door.
HP-UX 11 or later
HP-UX uses a named pipe called /dev/log that is padded to 2048 bytes, for example, source s_hp-ux {pipe ("/dev/log" pad-size(2048)}.
AIX 5.2 and 5.3
A SOCK_STREAM or SOCK_DGRAM unix socket called /dev/log.
Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using the SOCK_STREAM style — as used under Linux — is called unix-stream.
Example: Source statement on a Linux based operating system
The following source statement collects the following log messages:
internal(): Messages generated by syslog-ng.
network(transport(“udp”)): Messages arriving to the 514/UDP port of any interface of the host.
unix-dgram("/dev/log");: Messages arriving to the /dev/log socket.
The logging module is stored in the <prefix><module> name-value pair, for example: .radarr.module => ImportListSyncService.
You can modify the prefix with the prefix() option.
This driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
7.3 - Collect native macOS system logs
Starting with version 4.6.0, AxoSyslog can collect logs on macOS using its native OSLog framework using the darwin-oslog() and darwin-oslog-stream() source drivers.
darwin-oslog(): This source builds on the native OSLog framework, and replaces the earlier file-source based solution.
This source is based on the native OSLog Framework to read logs from the local store of the unified logging system on darwin OSes. The AxoSyslog system() source automatically uses this new source on darwin platforms if the darwinosl plugin is available. This plugin is available only on macOS 10.15 Catalina and above, the first version that has the OSLog API.
Note that the persistent OSLog store usually keeps about 7 days of logs on disk.
The darwin-oslog() source has the following options:
Description: By default, AxoSyslog continues to read the logs from the last remembered position after a restart. If this option is set to yes, it will always start reading from the end or beginning of the available log list (depending on the setting of the go-reverse() option).
fetch-delay()
Type:
integer
Default:
10000
Description: Controls the time AxoSyslog waits between reading and sending log messages. This is a fraction of a second, where wait_time = 1 second / n, so n=1 means that only about 1 log is sent in each second, and n=1000000 means only 1 microsecond is the delay between read/write attempts. The maximal value of this parameter is 1000000. Note that increasing the value of this parameter (thus lowering delay time) can increase log feed performance, but at the same time could increase system load.
fetch-retry-delay()
Type:
integer
Default:
1
Description: Controls how many seconds AxoSyslog waits before trying to check for new logs if there were no more logs to read the last time.
go-reverse()
Type:
boolean
Default:
no
Description: Set it to yes to process the logs in a reverse order (from latest to oldest).
log-fetch-limit()
Type:
integer
Default:
0 (no limit)
Warning
This option is currently disabled because of an OSLog API bug.
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
max-bookmark-distance()
Type:
integer
Default:
0 (no limit) [seconds]
Description: The maximum distance in seconds that a bookmark can point backwards. That is, if AxoSyslog was stopped for 10 minutes and max-bookmark-distance() is set to 60, then AxoSyslog will start reading the logs from 60 seconds before the startup, losing 9 minutes of logs.
read-old-records()
Type:
boolean
Default:
no
Description: If set to yes, AxoSyslog starts reading logs from the oldest available log when it’s first started on a system, or if there are no bookmarks for some reason
darwin-oslog-stream()
This source is a wrapper around the OS command line log stream command that provides a live log stream feed. Unlike the darwin-oslog() source, the live stream can contain non-persistent log events as well. This might result in a large number of log events every second.
This source has only one option.
params()
Type:
string
Default:
--type log --type trace --level info --level debug
Description: A string that can contain all the possible params the macOS log tool can accept. The source uses the “–style” internally (defaults to ndjson), so use templates or rewrite rules to format the final output. Use the def-osl-stream-params string to reference the default values when extending them with your own.
For a full reference, see the output of the log --help stream and man log commands.
7.4 - default-network-drivers: Receive and parse common syslog messages
The default-network-drivers() source is a special source that uses multiple source drivers to receive and parse several different types of syslog messages from the network. Available in version 3.16 and later.
To use the default-network-drivers() source, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
Also, make sure that your SELinux, AppArmor, and firewall settings permit AxoSyslog to access the ports where you want to receive messages, and that no other application is using these ports. By default, the default-network-drivers() source accepts messages on the following ports:
514, both TCP and UDP, for RFC3164 (BSD-syslog) formatted traffic
601 TCP, for RFC5424 (IETF-syslog) formatted traffic
6514 TCP, for TLS-encrypted traffic
In addition to receiving messages on different ports and in different formats, this source tries to parse the messages automatically. If successful, it sets the ${.app.name} name-value pair to the name of the application that sent the log message. Currently it uses the following procedures.
Warning
If you do not configure the TLS keys to show to the clients, AxoSyslog cannot accept encrypted connections. The application starts and listens on TCP:6514, and can receive messages on other ports, but will display a warning messages about missing keys.
Parsing RFC3164-formatted messages
For RFC3164-formatted messages (that is, messages received on the ports set in options udp-port() and tcp-port() which default to port 514), AxoSyslog attempts to use the following parsers. If a parser cannot parse the message, it passes the original message to the next parser.
The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving AxoSyslog component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first AxoSyslog component in your infrastructure has received it, which is important if you want to forward a message for example, to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.
Otherwise, apply the application adapters if the message was sent from an application that already has a specific parser in AxoSyslog (for example, Splunk Common Information Model (CIM), iptables, or sudo).
Parsing RFC5424-formatted messages
For RFC5424-formatted messages (that is, messages received on the ports set in options rfc5424-tls-port() and rfc5424-tcp-port(), which default to port 601 and 6514), AxoSyslog parses the message according to RFC5424, then attempts apply the application adapters if the message was sent from an application that already has a specific parser in AxoSyslog (for example, Splunk Common Information Model (CIM), iptables, or sudo).
Example: Using the default-network-drivers() driver
The following example uses only the default settings.
source s_network { default-network-drivers();};
The following example can receive TLS-encrypted connections on the default port (port 6514).
The systemd-journal() driver has the following options.
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
max-connections()
Type:
number
Default:
10
Description: Specifies the maximum number of simultaneous connections.
Note that the total number of connections the default-network-drivers() source can use is 3*max-connections(), because this value applies to the network(tcp), syslog(tcp), and syslog(tls) connections individually.
rfc5424-tcp-port()
Type:
number
Default:
601
Description: The TCP port number where the default-network-drivers() source receives RFC5424-formatted (IETF-syslog) messages.
rfc5424-tls-port()
Type:
number
Default:
6514
Description: The TCP port number where the default-network-drivers() source receives RFC5424-formatted (IETF-syslog), TLS-encrypted messages.
Warning
To receive messages using a TLS-encrypted connection, you must set the tls(key-file() cert-file()) options of the default-network-drivers() source. For example:
Description: The TCP port number where the default-network-drivers() source receives RFC3164-formatted (BSD-syslog) messages.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
udp-port()
Type:
number
Default:
514
Description: The UDP port number where the default-network-drivers() source receives RFC3164-formatted (BSD-syslog) messages.
7.5 - internal: Collect internal messages
All messages generated internally by AxoSyslog use this special source. To collect warnings, errors and notices from AxoSyslog itself, include this source in one of your source statements.
internal()
The AxoSyslog application will issue a warning upon startup if none of the defined log paths reference this driver.
Example: Using the internal() driver
source s_local { internal();};
The AxoSyslog application sends the following message types from the internal() source:
info: Priority value: info (6), Facility value: syslog (5)
7.5.1 - internal() source options
The internal() driver has the following options:
host-override()
Type:
string
Default:
Description: Replaces the ${HOST} part of the message with the parameter string.
log-iw-size()
Type:
number
Default:
100
Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.
normalize-hostnames()
Accepted values:
yes, no
Default:
no
Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
use-fqdn()
Type:
yes or no
Default:
no
Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
The AxoSyslog application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When AxoSyslog is restarted, it records the position of the last sent log message in the /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist file, and continues to send messages from this position after the restart.
If the message does not have a proper syslog header, AxoSyslog treats messages received from files as sent by the kern facility. Use the default-facility() and default-priority() options in the source definition to assign a different facility if needed.
7.6.1 - Notes on reading kernel messages
Note the following points when reading kernel messages on various platforms.
The kernel usually sends log messages to a special file (/dev/kmsg on BSDs, /proc/kmsg on Linux). The file() driver reads log messages from such files. The AxoSyslog application can periodically check the file for new log messages if the follow-freq() option is set.
On Linux, the klogd daemon can be used in addition to AxoSyslog to read kernel messages and forward them to syslog-ng. klogd used to preprocess kernel messages to resolve symbols and so on, but as this is deprecated by ksymoops there is really no point in running both klogd and AxoSyslog in parallel. Also note that running two processes reading /proc/kmsg at the same time might result in dead-locks.
When using AxoSyslog to read messages from the /proc/kmsg file, AxoSyslog automatically disables the follow-freq() parameter to avoid blocking the file.
To read the kernel messages on HP-UX platforms, use the following options in the source statement:
Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.
default-priority()
Type:
priority string
Default:
Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).
encoding()
Type:
string
Default:
Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
follow-freq()
Type:
number
Default:
1
Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
log-fetch-limit()
Type:
number
Default:
100
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
10000
Description: The size of the initial window, this value is used during flow control. Make sure that log-iw-size() is larger than the value of log-fetch-limit().
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
log-prefix() (DEPRECATED)
Type:
string
Default:
Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.
Note
This option is deprecated. Use program-override instead.
multi-line-garbage()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.
When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.
To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.
Warning
If the multi-line-garbage() option is set, AxoSyslog discards lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix().
Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:
indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.
prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.
prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.
The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.
smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.
smart mode is available in AxoSyslog version 4.2 and newer.
The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
multi-line-prefix()
Type:
regular expression starting with the ^ character
Default:
empty string
Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
Example: Processing Tomcat logs
The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.
2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.net.BindException: Address already in use null:8080
at org.apache.catalina.connector.Connector.start(Connector.java:1138) at org.apache.catalina.core.StandardService.start(StandardService.java:531) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177) 2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
INFO: Server startup in 1206 ms
2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:
Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.
multi-line-suffix()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.
To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
7.7 - wildcard-file: Collect messages from multiple text files
The wildcard-file() source collects log messages from multiple plain-text files from multiple directories. The wildcard-file() source is available in AxoSyslog version 3.10 and later.
The AxoSyslog application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When AxoSyslog is restarted, it records the position of the last sent log message in the persist file, and continues to send messages from this position after the restart. The location of the persist file depends on the package you installed AxoSyslog from, typically it is /var/lib/syslog-ng/syslog-ng.persist or /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist.
You can use the * and ? wildcard characters in the filename (the filename-pattern() option), but not in the path (the base-dir() option).
If you use multiple wildcard-file() sources in your configuration, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.
When using wildcards, AxoSyslog monitors every matching file (up to the limit set in the max-files() option), and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because AxoSyslog uses the inotify feature of the kernel. Set the max-files() option at least to the number of files you want to monitor. If the wildcard-file source matches more files than the value of the max-files() option, it is random which files will AxoSyslog actually monitor. The default value of max-files() is 100.
If the message does not have a proper syslog header, AxoSyslog treats messages received from files as sent by the user facility. Use the default-facility() and default-priority() options in the source definition to assign a different facility if needed.
For every message that AxoSyslog reads from the source files, the path and name of the file is available in the ${FILE_NAME} macro.
Required parameters: base-dir(), filename-pattern(). For the list of available optional parameters, see wildcard-file() source options.
Example: Using the wildcard-file() driver
The following example monitors every file with the .log extension in the /var/log directory for log messages.
The wildcard-file() driver has the following options:
base-dir()
Type:
path without filename
Default:
Description: The path to the directory that contains the log files to monitor, for example, base-dir("/var/log"). To monitor also the subdirectories of the base directory, use the recursive(yes) option. For details, see recursive().
Warning
If you use multiple wildcard-file() sources in your configuration, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.
Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.
default-priority()
Type:
priority string
Default:
Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).
encoding()
Type:
string
Default:
Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.
filename-pattern()
Type:
filename without path
Default:
Description: The filename to read messages from, without the path. You can use the * and ? wildcard characters, without regular expression and character range support. You cannot use the * and ? literally in the pattern.
For example, filename-pattern("*.log") matches the syslog.log and auth.log files, but does not match the access_log file. The filename-pattern("*log") pattern matches all three.
* matches an arbitrary string, including an empty string
? matches an arbitrary character
Warning
If you use multiple wildcard-file() sources in your configuration, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
follow-freq()
Type:
number
Default:
1
Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
log-fetch-limit()
Type:
number
Default:
100
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
10000
Description: The size of the initial window, this value is used during flow control. Make sure that log-iw-size() is larger than the value of log-fetch-limit().
When using wildcards in the filenames, AxoSyslog attempts to read log-fetch-limit() number of messages from each file. For optimal performance, make sure that log-iw-size() is greater than log-fetch-limit()*max-files(). Note that to avoid performance problems, if log-iw-size()/max-files() is smaller than 100, AxoSyslog automatically sets log-iw-size() to max-files()*100.
Example: Initial window size of file sources
If log-fetch-limit() is 100, and your wildcard file source has 200 files, then log-iw-size() should be at least 20000.
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
log-prefix() (DEPRECATED)
Type:
string
Default:
Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.
Note
This option is deprecated. Use program-override instead.
max-files()
Type:
integer
Default:
100
Description: Limits the number of files that the wildcard-file source monitors.
When using wildcards, AxoSyslog monitors every matching file (up to the limit set in the max-files() option), and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because AxoSyslog uses the inotify feature of the kernel. Set the max-files() option at least to the number of files you want to monitor. If the wildcard-file source matches more files than the value of the max-files() option, it is random which files will AxoSyslog actually monitor. The default value of max-files() is 100.
monitor-method()
Type:
`auto
Default:
auto
Description: If the platform supports inotify, AxoSyslog uses it automatically to detect changes to the source files. If inotify is not available, AxoSyslog polls the files as set in the follow-freq() option. To force AxoSyslog poll the files even if inotify is available, set this option to poll.
multi-line-garbage()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.
When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.
To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.
Warning
If the multi-line-garbage() option is set, AxoSyslog discards lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix().
Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:
indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.
prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.
prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.
The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.
smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.
smart mode is available in AxoSyslog version 4.2 and newer.
The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
multi-line-prefix()
Type:
regular expression starting with the ^ character
Default:
empty string
Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
Example: Processing Tomcat logs
The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.
2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.net.BindException: Address already in use null:8080
at org.apache.catalina.connector.Connector.start(Connector.java:1138) at org.apache.catalina.core.StandardService.start(StandardService.java:531) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177) 2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
INFO: Server startup in 1206 ms
2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:
Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.
multi-line-suffix()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.
To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
recursive()
Type:
`yes
Default:
no
Description: When enabled, AxoSyslog monitors every subdirectory of the path set in the base-dir() option, and reads log messages from files with matching filenames. The recursive option can be used together with wildcards in the filename.
Warning
If you use multiple wildcard-file() sources in your configuration, make sure that the files and folders that match the wildcards do not overlap. That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources can lead to data loss.
Example: Monitoring multiple directories
The following example reads files having the .log extension from the /var/log/ directory and its subdirectories, including for example, the /var/log/apt/history.log file.
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
7.8 - Hypr Audit Trail and Hypr App Audit Trail
Starting with version 4.2.0, AxoSyslog can fetch events from the Hypr REST API using the hypr-audit-trail() and hypr-app-audit-trail() source drivers.
hypr-audit-trail(): is a source driver that pulls messages from the Hypr API, associated to any RP Application ID.
hypr-app-audit-trail(): is a source driver that pulls messages from the Hypr API, but only those associated to a specific RP Application ID.
Hypr Audit Trail
The hypr-audit-trail() source queries the Hypr API for the list of potential applications at startup, then monitors the audit trail for each of the detected applications.
Note: Applications that are registered after syslog-ng is started are not recognized.
To start following those audit trails, you must restart syslog-ng.
source s_hypr { hypr-audit-trail( url('https://<custom domain>.hypr.com') bearer-token('<base64 encoded bearer token>') page-size(<number of results to return in a single page>) initial-hours(<number of hours to search backward on initial fetch>) application-skip-list('HYPRDefaultApplication', 'HYPRDefaultWorkstationApplication') log-level('INFO') flags(<optional flags passed to the source>) ignore-persistence(<yes/no>));};
Available options:
url(): custom URL for Hypr API access ('https://<custom domain>.hypr.com')
bearer-token(): base64 encoded authentication token from Hypr
page-size(): number of results to return in a single page (optional - defaults to 100)
initial-hours(): number of hours to search backward on initial fetch (optional - defaults to 4)
application-skip-list(): list of rpAppIds not to retrieve from Hypr (optional - defaults to 'HYPRDefaultApplication', 'HYPRDefaultWorkstationApplication')
log-level(): logging level, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" (optional - defaults to "INFO")
flags(): flags passed to the source, can be used for example to disable message parsing with flags(no-parse) (optional - defaults to empty)
ignore-persistence(): ignores the saved value in the persist file, and starts querying from the current time (optional - defaults to no)
Hypr App Audit Trail
The hypr-app-audit-trail() monitors the audit trail for one specific RP Application ID. This driver requires the rp-app-id() parameter in order to operate.
Available options:
url(): custom URL for Hypr API access ('https://<custom domain>.hypr.com')
bearer-token(): base64 encoded authentication token from Hypr
rp-app-id(): the RP Application ID for the application to monitor
page-size(): number of results to return in a single page (optional - defaults to 100)
initial-hours(): number of hours to search backward on initial fetch (optional - defaults to 4)
log-level(): logging level, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" (optional - defaults to "INFO")
flags(): flags passed to the source, can be used for example to disable message parsing with flags(no-parse) (optional - defaults to empty)
ignore-persistence(): ignores the saved value in the persist file, and starts querying from the current time (optional - defaults to no)
Acknowledgements
This documentation page is based on the README file of the hypr-audit-trail() source, written by Dan Elder.
7.9 - Jellyfin logs
Starting with version 4.7.0, you can use the jellyfin() source to read Jellyfin logs from its log file output.
This driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
7.10 - linux-audit: Collect messages from Linux audit logs
It reads and automatically parses the Linux audit logs. You can override the file name using the filename() parameter and the prefix for the created name-value pairs using the prefix() parameter. Any additional parameters are passed to the file source.
Note
Most recent Linux distributions enable Security-Enhanced Linux (SELinux) or AppArmor as a security measure. If enabled, these technologies might disable access to the Linux Audit log file by default. Consult their manuals to enable Linux Audit log access for AxoSyslog.
Description: The log file of linux-audit. The AxoSyslog application reads the Linux audit logs from this file.
prefix()
Synopsis:
prefix()
Default:
.auditd.
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. Note that if you use an empty prefix (prefix("")) or one starting with a dot, AxoSyslog might replace the original value of an existing macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
7.11 - kubernetes: Collect and parse the Kubernetes CRI (Container Runtime Interface) format
The kubernetes() source collects container logs managed by the Kubelet. It reads plain-text and JSON-formatted container logs (as described in the Container Runtime Interface (CRI) design proposal), for example, from the /var/log/containers or /var/log/pods files, and enriches them with various metadata retrieved from the Kubernetes API.
The kubernetes() source is available in AxoSyslog version 3.37 and later.
By default, it reads the /var/log/containers folder and extracts:
the log content, and
Kubernetes metadata, for example, namespace, pod, and container information.
The Kubernetes-related metadata is available in name-value pairs with the .k8s. prefix. The following table shows the retrieved metadata and their source:
syslog-ng name-value pair
source
.k8s.namespace_name
Container log file name.
.k8s.pod_name
Container log file name.
.k8s.pod_uuid
Container log file name or python kubernetes.client.CoreV1Api.
.k8s.container_name
Container log file name or python kubernetes.client.CoreV1Api.
.k8s.container_id
Container log file name.
.k8s.container_image
python kubernetes.client.CoreV1Api.
.k8s.container_hash
python kubernetes.client.CoreV1Api.
.k8s.docker_id
python kubernetes.client.CoreV1Api.
.k8s.labels.*
python kubernetes.client.CoreV1Api.
.k8s.annotations.*
python kubernetes.client.CoreV1Api.
Declaration
kubernetes( base-dir("<pathname>"));
7.11.1 - kubernetes() source options
The kubernetes() source has the following options:
base-dir()
Type:
path without filename
Default:
/var/log/containers
Description: The path to the directory that contains the log files, for example, base-dir("/var/log/pods").
cluster-name()
Type:
string
Default:
k8s
Description: The name of the Kubernetes cluster.
key-delimiter()
Type:
character
Default:
.
Description: The delimiter character to use when parsing flattened keys. Supports Only single characters.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
The prefix() option is optional and its default value is ".k8s.".
7.12 - mbox: Convert local email messages to log messages
Using the mbox() driver, AxoSyslog can read email messages from local mbox files, and convert them to multiline log messages.
This driver has only one required option, the filename of the mbox file. To use the mbox() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The mbox() driver is actually a reusable configuration snippet configured to read log messages using the file() driver. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the configuration snippet on GitHub.
Example: Using the mbox() driver
The following example reads the emails of the root user on the AxoSyslog host.
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
The rest of this chapter and its sections build on your familiarity with the MQTT protocol, the concept of client and broker entities, and how these entities function within an MQTT system.
Starting with AxoSyslog version 4.7, mqtt() source automatically sets the ${MQTT_TOPIC} name-value pair for the messages it receives. This is useful when the name of the topic contains MQTT wildcards ($, +, #). For example:
Example: Using the mqtt() source in your configuration
The following example illustrates an mqtt() source configured to fetch messages from the MQTT broker running on localhost:4444 using the test/test topic, and send them to the localhost:4445 address.
Using the current implementation of the mqtt() source has the following prerequisites:
Installing the eclipse-paho-mqtt-c library.
Note
The default package manager for some Linux operating systems contains the eclipse-paho-mqtt-c library, but depending on your OS, you may have to install the library manually. For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.
Having a broker entity in a functional MQTT system.
Note
In your configuration, you will specify the broker entity of your MQTT system in the address() option of your mqtt() source.
7.13.2 - Limitations to using the mqtt() source
Using the mqtt() source of AxoSyslog has the following limitations:
You can only use the mqtt() source with AxoSyslog version 3.35 or higher.
You cannot use the mqtt() source without installing the the eclipse-paho-mqtt-c library.
For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.
The current implementation of the mqtt() source supports versions 3.1 and 3.1.1 of the MQTT protocol
7.13.3 - Options of the mqtt() source
The mqtt() source has the following options.
Required options: address(), fallback-topic(), and topic().
address()
Type:
string
Default:
tcp://localhost:1883
Required:
yes
Description: Specifies the hostname or IP address, and the port number of the MQTT broker to which AxoSyslog will send the log messages.
Syntax: <protocol type>://<host>:<port>
client-id()
Type:
string
Default:
syslog-ng-source-{topic option}
Required:
no
Description: The client-id() is used to identify the client to the MQTT server, which stores session data for each client. The session data can contains information regarding which message has been sent, received. It is not possible to set the client-id() to an empty string. To always start a new session see the cleansession() option.
cleansession()
Type:
`yes
Default:
no
Description: This option instruments the MQTT broker to clean the session data when connecting. The session data contains information about which message was processed.
http-proxy()
Type:
URL
Default:
N/A
Description: Specifies HTTP/HTTPS proxy for WebSocket connections.
keep-alive()
Type:
positive integer number (in seconds)
Default:
60
Description: Specifies the number of seconds that AxoSyslog keeps the connection between the broker and clients open in case there is no message traffic. When keep-alive() number of seconds pass, the connection is terminated, and you have to reconnect.
On the MQTT side, the keep alive function provides a workaround method to access connections that are still open.
password()
Type:
string
Default:
N/A
Description: The password used to authenticate on the MQTT broker.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
qos()
Type:
number
Default:
`0`
Possible values:
`0` - at most once (the fastest option)
`1` - at least once (a much slower option than `0`)
`2` - exactly once (the slowest option)
Description: The Quality of Service (QoS) level in MQTT messaging is an agreement between sender and receiver on the guarantee of delivering a message.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
The following options are relevant for the mqtt()tls() block: ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), ssl-version(), use-system-cert-store().
topic()
Type:
string or template
Default:
N/A
Description: Required option. Specifies the MQTT topic.
username()
Type:
string
Default:
N/A
Description: The username used to authenticate on the MQTT broker.
7.14 - network: Collect messages using the RFC3164 protocol (network() driver)
The network() source driver can receive syslog messages conforming to RFC3164 from the network using the TCP, TLS, and UDP networking protocols.
UDP is a simple datagram oriented protocol, which provides “best effort service” to transfer messages between hosts. It may lose messages, and no attempt is made to retransmit lost messages. The BSD-syslog protocol traditionally uses UDP.
Use UDP only if you have no other choice.
TCP provides connection-oriented service: the client and the server establish a connection, each message is acknowledged, and lost packets are resent. TCP can detect lost connections, and messages are lost, only if the TCP connection breaks. When a TCP connection is broken, messages that the client has sent but were not yet received on the server are lost.
The AxoSyslog application supports TLS (Transport Layer Security, also known as SSL) over TCP. For details, see Encrypting log messages with TLS.
Declaration:
network([options]);
By default, the network() driver binds to 0.0.0.0, meaning that it listens on every available IPV4 interface on the TCP/514 port. To limit accepted connections to only one interface, use the localip() parameter. To listen on IPv6 addresses, use the ip-protocol(6) option.
Example: Using the network() driver
Using only the default settings: listen on every available IPV4 interface on the TCP/514 port.
source s_network { network();};
UDP source listening on 192.168.1.1 (the default port for UDP is 514):
A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslog messages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses both the IETF-syslog message format and the protocol. For details, see syslog: Collect messages using the IETF-syslog protocol.
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
By default, the kernel chooses the receive socket for a specific UDP randomly based on the source IP/port of the sender. You can customize this algorithm using the Extended Berkeley Packet Filter (eBPF) plugin. The ebpf() option changes the kernel’s SO_REUSEPORT algorithm so that all messages are randomly placed into one of the UDP sockets. The decision which UDP socket buffer a datagram is placed is made for every datagram, and not once for every stream. This means that messages are perfectly load-balanced across your set of UDP sockets. While this resolves the imbalance between the sockets and results in perfect load balancing, you will lose ordering between messages from the same sender, which is the price to pay for increased throughput.
Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
threaded: The threaded flag enables multithreading for the source. For details on multithreading, see Multithreading and scaling.
Note
The syslog source uses multiple threads only if the source uses the tls or tcp transport protocols.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Replaces the ${HOST} part of the message with the parameter string.
interface()
Type:
string
Default:
None
Description: Bind to the specified interface instead of an IP address. Available in 3.19 and later.
ip() or localip()
Type:
string
Default:
0.0.0.0
Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.
If you specify a multicast bind address and use the udp transport, AxoSyslog automatically joins the necessary multicast group. TCP does not support multicasting.
ip-protocol()
Type:
number
Default:
4
Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).
Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).
For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.
Description: Specifies the Type-of-Service value of outgoing packets.
ip-ttl()
Type:
number
Default:
0
Description: Specifies the Time-To-Live value of outgoing packets.
keep-alive()
Type:
yes or no
Default:
yes
Description: Specifies whether connections to sources should be closed when AxoSyslog is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the AxoSyslog connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.
keep-hostname()
Type:
yes or no
Default:
no
Description: Enable or disable hostname rewriting.
If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.
If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.
Note
If the log message does not contain a hostname in its HOST field, AxoSyslog automatically adds a hostname to the message.
For messages received from the network, this hostname is the address of the host that sent the message (this means the address of the last hop if the message was transferred via a relay).
For messages received from the local host, AxoSyslog adds the name of the host.
This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
When relaying messages, enable this option on the AxoSyslog server and also on every relay, otherwise AxoSyslog will treat incoming messages as if they were sent by the last relay.
keep-timestamp()
Type:
yes or no
Default:
yes
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
listen-backlog()
Type:
integer
Default:
256
Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
100
Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.
If the max-connections() option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection. For optimal performance when receiving messages from AxoSyslog clients, make sure that the window size is larger than the flush-lines() option set in the destination of your clients.
Example: Initial window size of a connection
If log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
max-connections()
Type:
number
Default:
10
Description: Specifies the maximum number of simultaneous connections.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
port() or localport()
Type:
number
Default:
In case of TCP transport: 514
In case of UDP transport: 514
Description: The port number to bind to.
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
so-broadcast()
Type:
yes or no
Default:
no
Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.
so-keepalive()
Type:
yes or no
Default:
no
Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-rcvbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.
Warning
When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the AxoSyslog server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf() option of the source is increased. In this cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.
As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152 bytes.
so-reuseport()
Type:
yes or no
Default:
no
Description: Enables SO_REUSEPORT on systems that support it. When enabled, the kernel allows multiple UDP sockets to be bound to the same port, and the kernel load-balances incoming UDP datagrams to the sockets. The sockets are distributed based on the hash of (srcip, dstip, srcport, dstport), so the same listener should be receiving packets from the same endpoint. For example:
Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-sndbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
Description: Specifies the protocol used to receive messages from the source.
For detailed information about how AxoSyslog supports the proxied-tcp, the proxied-tls, and the proxied-tls-passthrough parameters, see Proxy Protocol support.
text-with-nuls: Allows embedded NUL characters in the message from a TCP source, that is, AxoSyslog will not delimiter the incoming messages on NUL characters, only on newline characters (contrary to tcp transport, which splits the incoming log on newline characters and NUL characters).
Note
The AxoSyslog application does not support embedded NUL characters everywhere, so it is recommended that you also use flags(no-multi-line) that causes NUL characters to be replaced by space.
Warning
When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the AxoSyslog server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf() option of the source is increased. In this cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.
As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152 bytes.
trim-large-messages()
Type:
yes
Default:
Use the global trim-large-messages() option, which defaults to no.
Description: Determines what AxoSyslog does with incoming log messages that are received using the IETF-syslog protocol using the syslog() driver, and are longer than the value of log-msg-size(). Other drivers ignore the trim-large-messages() option.
If set to no, AxoSyslog drops the incoming log message.
If set to yes, AxoSyslog trims the incoming log message to the size set in log-msg-size(), and adds the trimmed tag to the message. The rest of the message is dropped. You can use the tag to filter on such messages.
filter f_trimmed { tags("trimmed");};
If AxoSyslog trims a log message, it sends a debug-level log message to its internal() source.
As a result of trimming, a parser could fail to parse the trimmed message. For example, a trimmed JSON or XML message will not be valid JSON or XML.
Available in AxoSyslog version 3.21 and later.
Uses the value of the global option if not specified.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
use-dns()
Type:
yes, no, persist_only
Default:
yes
Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example, from /etc/hosts). The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
use-fqdn()
Type:
yes or no
Default:
no
Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.
Note
Set use-fqdn() to yes if you want to use the custom-domain() global option.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
7.14.2 - Proxy Protocol support
If you connect load balancers to your AxoSyslog application, AxoSyslog identifies every connection that is connected to the load balancers identically by default, regardless of the source IP or the source protocol. Essentially, the load balancer masks the source IP unless you enable Proxy Protocol support for your proxy TLS transport() to inject information about the original connection into the forwarded TCP session.
For further details about the working mechanism behind the Proxy Protocol support on AxoSyslog and the configuration details, see the following sections:
7.14.2.1 - The working mechanism behind the Proxy Protocol
This section describes how AxoSyslog supports the Proxy Protocol.
The working mechanism behind the Proxy Protocol
When using the Proxy Protocol during load balancing, AxoSyslog detects the information behind connections connected to the load balancer, then parses the injected information and adds the following macros to every message the comes through the connection later on:
PROXY_SRCIP (the source IP of the proxy)
PROXY_SRCPORT (the source port of the proxy)
PROXY_DSTIP (the destination IP of the proxy)
PROXY_DSTPORT (the destination port of the proxy)
Note
Consider the following about macros and headers:
When the proxy protocol header is PROXY UNKNOWN, no additional macros are added.
When AxoSyslog cannot parse a proxy protocol header, the connection is closed:
Originally, the driver supported version 1 of the PROXY protocol (TCP4, TCP6, and TLS connections). PROXY protocol v2 support is available in AxoSyslog version 4.1 and later.
7.14.2.2 - Proxy Protocol: configuration and output examples
This section provides information about enabling Proxy Protocol support in your network() source options, and an example configuration and output to illustrate how the Proxy Protocol method works in AxoSyslog.
Enabling Proxy Protocol support for your network() source options
Unless you enable Proxy Protocol support for your network() source, AxoSyslog identifies every connection that is connected to the load balancers identically by default, regardless of the source IP or the source protocol.
proxied-tls can be used in complex MITM (man in the middle) configurations, where the proxy header is sent encrypted within the same TLS session as the proxied messages.
When you enable Proxy Protocol support for your network() source, you can use the following configuration example with your AxoSyslog application.
Configuration
The following code sample illustrates how you can use the Proxy Protocol in your AxoSyslog configuration (using the transport() parameter set to proxied-tls-passthrough).
With this configuration, the Proxy Protocol method will perform injecting the information of the original connection into the forwarded TCP session, based on the working mechanism described in The working mechanism behind the Proxy Protocol.
The following example illustrates how the parsed macros will appear in the output.
Example: Output for the PROXY TCP4 192.168.1.1 10.10.0.1 1111 2222 input header
With the PROXY TCP4 192.168.1.1 10.10.0.1 1111 2222 input header, the output looks like this:
Note that the macros that AxoSyslog adds to the message appear in the output.
7.15 - nodejs: Receive JSON messages from nodejs applications
Using the nodejs() driver, AxoSyslog can receive application logs directly from nodejs applications that use the widespread Winston logging API. The AxoSyslog application automatically adds the .nodejs.winston. prefix to the name of the fields the extracted from the message.
To use the nodejs() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The nodejs() driver is actually a reusable configuration snippet configured to receive log messages using the network() driver, and process its JSON contents. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the nodejs configuration snippet on GitHub.
Example: Using the nodejs() driver
The following example uses the default settings of the driver, listening for messages on port 9003 of every IP address of the AxoSyslog host.
@include "scl.conf"source apps { nodejs();};
The following example listens only on IP address 192.168.1.1, port 9999.
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.
If you specify a multicast bind address and use the udp transport, AxoSyslog automatically joins the necessary multicast group. TCP does not support multicasting.
port() or localport()
Type:
number
Default:
9003
Description: The port number to bind to.
7.16 - osquery: Collect and parse osquery result logs
The osquery application allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. You can make queries on demand, and also schedule them to run regularly.
The osquery() source of AxoSyslog allows you read the results of periodical osquery queries (from the /var/log/osquery/osqueryd.results.log file) and automatically parse the messages (if you want to use AxoSyslog to send log messages to osquery, read this blogpost). For example, you can:
Create filters from the fields of the messages.
Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on).
Send the messages to a central location, for example, to Elasticsearch, directly from AxoSyslog.
The AxoSyslog application automatically adds the .osquery. prefix to the name of the fields the extracted from the message.
The osquery() source is available in AxoSyslog version 3.10 and later.
Prerequisites:
To use the osquery() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
AxoSyslog must be compiled with JSON-support enabled.
The osquery() driver is actually a reusable configuration snippet configured to read the osquery log file using the file() driver, and process its JSON contents. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
Example: Using the osquery() driver with the default settings
The following AxoSyslog configuration sample uses the default settings of the driver, reading osquery result logs from the /var/log/osquery/osqueryd.results.log file, and writes the log messages generated from the traps into a file.
Example: Using the osquery() driver with custom configuration
The following AxoSyslog configuration sample reads osquery result logs from the /tmp/osquery_input.log file, and writes the log messages generated from the traps into a file. Using the format-json template, the outgoing message will be a well-formed JSON message.
Description: The log file of osquery that stores the results of periodic queries. The AxoSyslog application reads the messages from this file.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
Default value:
.osquery. option.
7.17 - Receive logs, metrics, and traces from OpenTelemetry
The following example receives OpenTelemetry data and forwards it to an OpenTelemetry receiver. Note that by default, AxoSyslog doesn’t parse the fields of the incoming messages into name-value pairs, but are only available for forwarding using the opentelemetry() destination. To parse the fields into name-value pairs, use the opentelemetry() parser.
Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.
tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.
channel-args()
Type:
See description
Default:
-
Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:
Description: Configures the maximal number of in-flight gRPC requests per worker. Setting this value in the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater than or equal to the number of clients, but this can increase the memory usage.
log-fetch-limit()
Type:
number
Default:
100
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
port()
The port number to receive incoming connections. Default value: 4317
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to process the messages of the source. Increasing the number of worker threads can drastically improve the performance of the source.
7.18 - pacct: Collect process accounting logs on Linux
Starting with version 3.2, AxoSyslog can collect process accounting logs on Linux systems.Process accounting is the method of recording and summarizing commands executed on Linux, for example, the commands being run, the user who executed the command, CPU time used by the process, exit code, and so on. When process accounting (also called pacct) is enabled on a system, the kernel writes accounting records to the /var/log/account/pacct file (might vary between different Linux distributions).
To use the pacct() driver, the following conditions must be met:
The AxoSyslog application must be compiled with the --enable-pacct option. Execute the syslog-ng -V command to check if your binary supports process accounting.
The pacctformat plugin must be loaded. By default, AxoSyslog automatically loads the available modules.
The scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
Process accounting must be running on the host. You can enable it with the accton command.
The pacct() driver parses the fields of the accounting logs and transforms them into name-value pairs. The fields are defined in the manual page of the accounting log file (man acct), AxoSyslog prepends every field with the .pacct. prefix. For example, the ac_uid field that contains the id of the user who started the process will be available under the $.pacct.ac_uid name. These can be used as macros in templates, in filters to select specific messages, and so on.
To use the pacct() driver, use the following syntax.
The pacct() driver is actually a reusable configuration snippet configured to handle Linux accounting logs. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the pacct configuration snippet on GitHub.
7.18.1 - pacct() options
The pacct() driver has the following options:
file()
Type:
filename with path
Default:
/var/log/account/pacct
Description: The file where the process accounting logs are stored — AxoSyslog reads accounting messages from this file.
follow-freq()
Type:
number
Default:
1
Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Starting with version 4.6.0, AxoSyslog can collect logs of the Pi-hole FTL (Faster Than Light) application.
source s_pihole_ftl { pihole-ftl();};
By default, the source reads the /var/log/pihole/FTL.log file. If the root directory of your Pi-hole installation is different, specify the directory where the FTL.log file is with the dir() option. You can find the root log directory by selecting Tools > Pi-hole diagnosis in the Pi-hole application. Otherwise, the pihole-ftl() source has the same parameters as the file() source.
The pihole-ftl() driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
7.20 - pipe: Collect messages from named pipes
The pipe driver opens a named pipe with the specified name and listens for messages. It is used as the native message delivery protocol on HP-UX.
The pipe driver has a single required parameter, specifying the filename of the pipe to open. For the list of available optional parameters, see pipe() source options.
Declaration:
pipe(filename);
Note
As of version 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.
Pipe is very similar to the file() driver, but there are a few differences, for example, pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg.
Warning
It is not recommended to use pipe() on anything else than real pipes.
By default, AxoSyslog uses the flags(no-hostname) option for pipes, meaning that AxoSyslog assumes that the log messages received from the pipe do not contain the hostname field. If your messages do contain the hostname field, use flags(expect-hostname).
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
follow-freq()
Type:
number
Default:
1
Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
log-fetch-limit()
Type:
number
Default:
100
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
100
Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
log-prefix() (DEPRECATED)
Type:
string
Default:
Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.
Note
This option is deprecated. Use program-override instead.
multi-line-garbage()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.
When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.
To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.
Warning
If the multi-line-garbage() option is set, AxoSyslog discards lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix().
Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:
indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.
prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.
prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.
The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.
smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.
smart mode is available in AxoSyslog version 4.2 and newer.
The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
multi-line-prefix()
Type:
regular expression starting with the ^ character
Default:
empty string
Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
Example: Processing Tomcat logs
The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.
2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.net.BindException: Address already in use null:8080
at org.apache.catalina.connector.Connector.start(Connector.java:1138) at org.apache.catalina.core.StandardService.start(StandardService.java:531) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177) 2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
INFO: Server startup in 1206 ms
2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:
Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.
multi-line-suffix()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.
To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.
optional()
Type:
yes or no
Default:
Description: Instruct AxoSyslog to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
7.21 - program: Receive messages from external applications
The program driver starts an external application and reads messages from the standard output (stdout) of the application. It is mainly useful to receive log messages from daemons that accept incoming messages and convert them to log messages.
The program driver has a single required parameter, specifying the name of the application to start.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
log-fetch-limit()
Type:
number
Default:
100
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
inherit-environment()
Type:
yes
Default:
yes
Description: By default, when program() starts an external application or script, it inherits the entire environment of the parent process (that is, AxoSyslog). Use inherit-environment(no) to prevent this.
log-iw-size()
Type:
number
Default:
100
Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
log-prefix() (DEPRECATED)
Type:
string
Default:
Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.
Note
This option is deprecated. Use program-override instead.
optional()
Type:
yes or no
Default:
Description: Instruct AxoSyslog to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
program()
Type:
filename with path
Default:
Description: The name of the application to start and read messages from.
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
The Python source allows you to write your own source in Python. You can import external Python modules to receive or fetch the messages. Since many services have a Python library, the Python source makes integrating AxoSyslog very easy and quick.
You can write two different type of sources in Python:
Server-style sources that receives messages. Write server-style sources if you want to use an event-loop based, nonblocking server framework in Python, or if you want to implement a custom loop.
Fetcher-style sources that actively fetch messages. In general, write fetcher-style sources (for example, when using simple blocking APIs), unless you explicitly need a server-style source.
The following points apply to using Python blocks in AxoSyslog in general:
Python parsers and template functions are available in AxoSyslog version 3.10 and later.
Python destinations and sources are available in AxoSyslog version 3.18 and later.
Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).
The Python block must be a top-level block in the AxoSyslog configuration file.
If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.
In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".
The Python object is initiated every time when AxoSyslog is started or reloaded.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
The Python block can contain multiple Python functions.
Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.
Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.
Python error messages are available in the internal() source of AxoSyslog.
You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.
Note
Starting with 3.26, AxoSyslog assigns a persist name to Python sources and destinations. The persist name is generated from the class name. If you want to use the same Python class multiple times in your AxoSyslog configuration, add a unique persist-name() to each source or destination, otherwise AxoSyslog will not start. For example:
Alternatively, you can include the following line in the Python package: @staticmethod generate_persist_name. For example:
from syslogng import LogSource
class PyNetworSource(LogSource):
@staticmethod
def generate_persist_name(options):
return options["port"] def run(self):
pass
def request_exit(self):
pass
Declaration:
Python sources consist of two parts. The first is a AxoSyslog source object that you define in your AxoSyslog configuration and use in the log path. This object references a Python class, which is the second part of the Python source. The Python class receives or fetches the log messages, and can do virtually anything that you can code in Python. You can either embed the Python class into your AxoSyslog configuration file, or store it in an external Python file.
source <name_of_the_python_source>{ python( class("<name_of_the_python_class_executed_by_the_source>") options("option1""value1",
"option2""value2"));}; python { from syslogng import LogSource
from syslogng import LogMessage
class <name_of_the_python_class_executed_by_the_source>(LogSource):
def init(self, options): # optional print("init") print(options) self.exit = False
return True
def deinit(self): # optional print("deinit") def run(self): # mandatory print("run")while not self.exit:
# Must create a messagemsg= LogMessage("this is a log message") self.post_message(msg) def request_exit(self): # mandatory print("exit") self.exit = True
};
Methods of the python() source
Server-style Python sources must be inherited from the syslogng.LogSource class, and must implement at least the run and request_exit methods. Multiple inheritance is allowed, but only for pure Python super classes.
You can implement your own event loop, or integrate the event loop of an external framework or library, for example, KafkaConsumer, Flask, Twisted engine, and so on.
To post messages, call LogSource::post_message() method in the run method.
The AxoSyslog application initializes Python objects every time when it is started or reloaded. The init method is executed as part of the initialization. You can perform any initialization steps that are necessary for your source to work.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
When this method returns with False, AxoSyslog does not start. It can be used to check options and return False when they prevent the successful start of the source.
options: This optional argument contains the contents of the options() parameter of the AxoSyslog configuration object as a Python dictionary.
run(self) method (mandatory)
Use the run method to implement an event loop, or start a server framework or library. Create LogMessage instances in this method, and pass them to the log paths by calling LogSource::post_message().
Currently, run stops permanently if an unhandled exception happens.
The AxoSyslog application calls this method when AxoSyslog is shut down or restarted. The request_exit method must shut down the event loop or framework, so the run method can return gracefully. If you use blocking operations within the run() method, use request_exit() to interrupt those operations and set an exit flag, otherwise AxoSyslog is not able to stop. Note that AxoSyslog calls the request_exit method from a thread different from the source thread.
close_batch(self)
Closes the current source-side batch. Source-side batching helps AxoSyslog to effectively process a larger chunk of messages, instead of processing messages each message. For example, when feeding a destination queue and instead of taking a lock on the queue for every message (causing contention), we only take it once per batch.
The native drivers built into AxoSyslog typically close batches once every mainloop iteration, allowing a single iteration to process multiple messages. For instance, when receiving multiple messages in a single TCP datagram, all of those messages
can be processed as a part of the same batch.
In Python-based log sources, a batch will automatically be closed after every message posted via post_message(), except if self.auto_close_batches is set to False during initialization. In case self.auto_close_batches is set to False, the driver has to call close_batch() explicitly, preferably at a natural boundary between incoming batches of messages. A good example is when we retrieve several messages via the same HTTP REST call, then the right time to close the batch would be after the last message in the response is posted.
The deinit(self) method (optional)
This method is executed when AxoSyslog is stopped or reloaded. This method does not return a value.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
set_transport_name(self, name)
Set the transport name used to retrieve messages This function can be called to customize the ${TRANSPORT} name-value pair.
7.22.1 - Python LogMessage API
The LogMessage API allows you to create LogMessage objects in Python sources, parse syslog messages, and set the various fields of the log message.
LogMessage() method: Create log message objects
You can use the LogMessage() method to create a structured log message instance. For example:
from syslogng import LogMessage
msg= LogMessage()# Initialize an empty message with default values (recvd timestamp, rcptid, hostid, ...)msg= LogMessage("string or bytes-like object")# Initialize a message and set its ${MESSAGE} field to the specified argument
You can also explicitly set the different values of the log message. For example:
msg["MESSAGE"]="message" msg["HOST"]="hostname"
You can set certain special field (timestamp, priority) by using specific methods.
Note the following points when creating a log message:
When setting the hostname, AxoSyslog takes the following hostname-related options of the configuration into account: chain-hostnames(), keep-hostname(), use-dns(), and use-fqdn().
Python sources ignore the log-msg-size() option.
The AxoSyslog application accepts only one message from every LogSource::post_message() or fetch() call, batching is currently not supported. If your Python code accepts batches of messages, you must pass them to AxoSyslog one-by-one. Similarly, if you need to split messages in the source, you must do so in your Python code, and pass the messages separately.
Do not reuse or store LogMessage objects after posting (calling post_message()) or returning the message from fetch().
parse() method: Parse syslog messages
The parse() method allows you to parse incoming messages as syslog messages. By default, the parse() method attempts to parse the message as an IETF-syslog (RFC5424) log message. If that fails, it parses the log message as a BSD-syslog (RFC3164) log message. Note that AxoSyslog takes the parsing-related options of the configuration into account: flags(), keep-hostname(), recv-time-zone().
If keep-hostname() is set to no, AxoSyslog ignores the hostname set in the message, and uses the IP address of the AxoSyslog host as the hostname (to use the hostname instead of the IP address, set the use-dns() or use-fqdn() options in the Python source).
msg_ietf= LogMessage.parse('<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry', self.parse_options)msg_bsd= LogMessage.parse('<34>Oct 11 22:14:15 mymachine su: \'su root\' failed for lonvick on /dev/pts/8', self.parse_options)
set_pri() method
You can set the priority of the message with the set_pri() method.
msg.set_pri(165)
set_timestamp() method
You can use the set_timestamp() method to set the date and time of the log message.
timestamp= datetime.fromisoformat("2018-09-11T14:49:02.100+02:00") msg.set_timestamp(timestamp)# datetime object, includes timezone information
In Python 2, timezone information cannot be attached to the datetime instance without using an external library. The AxoSyslog represents naive datetime objects in UTC.
In Python 3, naive and timezone-aware datetime objects are both supported.
7.22.2 - python() and python-fetcher() source options
The python() and python-fetcher() drivers have the following options.
class()
Type:
string
Default:
N/A
Description: The name of the Python class that implements the source, for example:
python( class("MyPythonSource"));
If you want to store the Python code in an external Python file, the class() option must include the name of the Python file containing the class, without the path and the .py extension, for example:
Description: If the fetch method of a python-fetcher() source returns with the LogFetcher.FETCH_NO_DATA constant, the source waits fetch-no-data-delay() seconds before calling the fetch method again. If you want to call the fetch method sooner, set the fetch-no-data-delay() option to the number of seconds to wait before calling the fetch method.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
The flags and the hostname-related options (for example, use-dns) set in the configuration file influence the behavior of the LogMessage.parse() method of the Python source. They have no effect if you set the message or the hostname directly, without using LogMessage.parse().
keep-hostname()
Type:
yes or no
Default:
no
Description: Enable or disable hostname rewriting.
If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.
If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.
Note
If the log message does not contain a hostname in its HOST field, AxoSyslog automatically adds a hostname to the message.
For messages received from the network, this hostname is the address of the host that sent the message (this means the address of the last hop if the message was transferred via a relay).
For messages received from the local host, AxoSyslog adds the name of the host.
This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
When relaying messages, enable this option on the AxoSyslog server and also on every relay, otherwise AxoSyslog will treat incoming messages as if they were sent by the last relay.
log-iw-size()
Type:
number
Default:
100
Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.
loaders()
Type:
list of python modules
Default:
empty list
Description: The AxoSyslog application imports Python modules specified in this option, before importing the code of the Python class. This option has effect only when the Python class is provided in an external Python file. This option has no effect when the Python class is provided within the AxoSyslog configuration file (in a python{} block). You can use the loaders() option to modify the import mechanism that imports Python class. For example, that way you can use hy in your Python class.
python(class(usermodule.HyParser) loaders(hy))
options()
Type:
string
Default:
N/A
Description: This option allows you to pass custom values from the configuration file to the Python code. Enclose both the option names and their values in double-quotes. The Python code will receive these values during initialization as the options dictionary. For example, you can use this to set the IP address of the server from the configuration file, so it is not hard-coded in the Python object.
For example, you can refer to the value of the host field in the Python code as options["host"]. Note that the Python code receives the values as strings, so you might have to cast them to the type required, for example: int(options["port"])
Note
From version 3.27, AxoSyslog supports the arrow syntax for declaring custom Java and Python options. You can alternatively declare them using a similar syntax:
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
Note
Starting with 3.26, AxoSyslog assigns a persist name to Python sources and destinations. The persist name is generated from the class name. If you want to use the same Python class multiple times in your AxoSyslog configuration, add a unique persist-name() to each source or destination, otherwise AxoSyslog will not start. For example:
Alternatively, you can include the following line in the Python package: @staticmethod generate_persist_name. For example:
from syslogng import LogSource
class PyNetworSource(LogSource):
@staticmethod
def generate_persist_name(options):
return options["port"] def run(self):
pass
def request_exit(self):
pass
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-reopen()
Accepted values:
number [seconds]
Default:
1
Description: The time to wait in seconds before a dead connection is reestablished.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
Warning
This option is available only when using Python 3.
The Python source allows you to write your own source in Python. You can import external Python modules to receive or fetch the messages. Since many services have a Python library, the Python source makes integrating AxoSyslog very easy and quick.
You can write two different type of sources in Python:
Server-style sources that receives messages. Write server-style sources if you want to use an event-loop based, nonblocking server framework in Python, or if you want to implement a custom loop.
Fetcher-style sources that actively fetch messages. In general, write fetcher-style sources (for example, when using simple blocking APIs), unless you explicitly need a server-style source.
The following points apply to using Python blocks in AxoSyslog in general:
Python parsers and template functions are available in AxoSyslog version 3.10 and later.
Python destinations and sources are available in AxoSyslog version 3.18 and later.
Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).
The Python block must be a top-level block in the AxoSyslog configuration file.
If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.
In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".
The Python object is initiated every time when AxoSyslog is started or reloaded.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
The Python block can contain multiple Python functions.
Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.
Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.
Python error messages are available in the internal() source of AxoSyslog.
You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.
Declaration:
Python sources consist of two parts. The first is a AxoSyslog source object that you define in your AxoSyslog configuration and use in the log path. This object references a Python class, which is the second part of the Python source. The Python class receives or fetches the log messages, and can do virtually anything that you can code in Python. You can either embed the Python class into your AxoSyslog configuration file, or store it in an external Python file.
source <name_of_the_python_source>{ python-fetcher( class("<name_of_the_python_class_executed_by_the_source>"));}; python { from syslogng import LogFetcher
from syslogng import LogMessage
class <name_of_the_python_class_executed_by_the_source>(LogFetcher):
def init(self, options): # optional print("init") print(options)return True
def deinit(self): # optional print("deinit") def open(self): # optional print("open")return True
def fetch(self): # mandatory print("fetch")# return LogFetcher.FETCH_ERROR,# return LogFetcher.FETCH_NOT_CONNECTED,# return LogFetcher.FETCH_TRY_AGAIN,# return LogFetcher.FETCH_NO_DATA,return LogFetcher.FETCH_SUCCESS, msg
def request_exit(self):
print("request_exit")# If your fetching method is blocking, do something to break it# For example, if it reads a socket: socket.shutdown() def close(self): # optional print("close")};
Methods of the python-fetcher() source
Fetcher-style Python sources must be inherited from the syslogng.LogFetcher class, and must implement at least the fetch method. Multiple inheritance is allowed, but only for pure Python super classes.
For fetcher-style Python sources, AxoSyslog handles the event loop and the scheduling automatically. You can use simple blocking server/client libraries to receive or fetch logs.
You can retrieve messages using the fetch() method.
init(self, options) method (optional)
The AxoSyslog application initializes Python objects every time when it is started or reloaded. The init method is executed as part of the initialization. You can perform any initialization steps that are necessary for your source to work.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
When this method returns with False, AxoSyslog does not start. It can be used to check options and return False when they prevent the successful start of the source.
options: This optional argument contains the contents of the options() parameter of the AxoSyslog configuration object as a Python dictionary.
open(self) method (optional)
The open(self) method opens the resources required for the source, for example, it initiates a connection to the target service. It is called after init() when AxoSyslog is started or reloaded. If fetch() returns with an error, AxoSyslog calls the close() and open() methods before trying to fetch a new message.
If open() fails, it should return the False value. In this case, AxoSyslog retries it every time-reopen() seconds. By default, this is 1 second for Python sources and destinations, the value of time-reopen() is not inherited from the global option. For details, see Error handling in the python() destination.
fetch(self) method (mandatory)
Use the fetch method to fetch messages and pass them to the log paths.
The fetch method must return one of the following values:
LogFetcher.FETCH_ERROR: Fetching new messages failed, AxoSyslog calls the close and open methods.
LogFetcher.FETCH_NO_DATA: There was not any data available. The source waits before calling the fetch method again. The wait time is equal to time-reopen() by default, but you can override it by setting the fetch-no-data-delay() option in the source.
LogFetcher.FETCH_NOT_CONNECTED: Could not access the source, AxoSyslog calls the open method.
LogFetcher.FETCH_SUCCESS, msg: Post the message returned as the second argument.
LogFetcher.FETCH_TRY_AGAIN: The fetcher could not provide a message this time, but will make the source call the fetch method as soon as possible.
request_exit(self) method (optional)
If you use blocking operations within the fetch() method, use request_exit() to interrupt those operations (for example, to shut down a socket), otherwise AxoSyslog is not able to stop. Note that AxoSyslog calls the request_exit method from a thread different from the source thread.
close(self) method (optional)
Close the connection to the target service. Usually it is called right before deinit() when stopping or reloading AxoSyslog. It is also called when fecth() fails.
close_batch(self)
Closes the current source-side batch. Source-side batching helps AxoSyslog to effectively process a larger chunk of messages, instead of processing messages each message. For example, when feeding a destination queue and instead of taking a lock on the queue for every message (causing contention), we only take it once per batch.
The native drivers built into AxoSyslog typically close batches once every mainloop iteration, allowing a single iteration to process multiple messages. For instance, when receiving multiple messages in a single TCP datagram, all of those messages
can be processed as a part of the same batch.
In Python-based log sources, a batch will automatically be closed after every message posted via post_message(), except if self.auto_close_batches is set to False during initialization. In case self.auto_close_batches is set to False, the driver has to call close_batch() explicitly, preferably at a natural boundary between incoming batches of messages. A good example is when we retrieve several messages via the same HTTP REST call, then the right time to close the batch would be after the last message in the response is posted.
The deinit(self) method (optional)
This method is executed when AxoSyslog is stopped or reloaded. This method does not return a value.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
To configure the source, you only have to specify the root log directory of qBittorrent in the dir() parameter. You can find the root log directory by selecting Tools > Preferences > Behavior > Log file > Save path in the qBittorrent application. Otherwise, the qbittorrent() source has the same parameters as the file() source.
The qbittorrent() driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
7.25 - snmptrap: Read Net-SNMP traps
Using the snmptrap() source, you can read and parse the SNMP traps of the Net-SNMP’s snmptrapd application. AxoSyslog can read these traps from a log file, and extract their content into name-value pairs, making it easy to forward them as a structured log message (for example, in JSON format). The AxoSyslog application automatically adds the .snmp. prefix to the name of the fields the extracted from the message.
The snmptrap() source is available in AxoSyslog version 3.10 and later.
Limitations:
The snmptrap() source has only the options listed in snmptrap() source options. Other options commonly available in other source drivers are not supported.
In addition to traps, the log of snmptrapd may contain other messages (for example, daemon start/stop information, debug logs) as well. Currently AxoSyslog discards these messages.
The AxoSyslog application cannot resolve OIDs, you have to configure snmptrapd to do so. Note that because of a bug, if snmptrapd does not escape String values in the VarBindList if it can resolve an OID to a symbolic name. As a result, AxoSyslog cannot process traps that contain the = in the value of the string. To overcome this problem, disable resolving OIDs in snmptrapd. For details, see the documentation of snmptrapd.
The colon (:) character is commonly used in SNMP traps. However, this character cannot be used in the name of AxoSyslog macros (name-value pairs). Therefore, the AxoSyslog application automatically replaces all consecutive : characters with a single underscore (_) character. For example, you can reference the value of the NET-SNMP-EXAMPLES-MIB::netSnmpExampleString key using the ${NET-SNMP-EXAMPLES-MIB_netSnmpExampleString} macro.
Note that this affects only name-value pairs (macros). The generated message always contains the original name of the key.
Prerequisites:
Configure snmptrapd to log into a file.
If you use SMIv1 traps, include the following format string in the configuration file of snmptrapd:
Beacause of an snmptrapd bug, if you specify the filename in the configuration file with logOption, you must also specify another output as a command line argument (-Lf, -Ls). Otherwise, snmptrapd will not apply the the trap format.
To use the snmptrap() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
Example: Using the snmptrap() driver
A sample snmptrapd configuration:
authCommunity log,execute,net public
format1 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]: %N\n\t%W Trap (%q) Uptime: %#T\n%v\n outputOption s
Starting snmptrapd: snmptrapd -A -Lf /var/log/snmptrapd.log
Sending a sample V2 trap message: snmptrap -v2c -c public 127.0.0.1 666 NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification netSnmpExampleHeartbeatRate i 60 netSnmpExampleString s "string". From this trap, AxoSyslog receives the following input:
The following AxoSyslog configuration sample uses the default settings of the driver, reading SNMP traps from the /var/log/snmptrapd.log file, and writes the log messages generated from the traps into a file.
The snmptrap() driver has the following options. Only the filename() option is required, the others are optional.
filename()
Type:
path
Default:
Description: The log file of snmptrapd. The AxoSyslog application reads the traps from this file.
In addition to traps, the log of snmptrapd may contain other messages (for example, daemon start/stop information, debug logs) as well. Currently AxoSyslog discards these messages.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
Default value: .snmp. option.
set-message-macro()
Type:
yes
Default:
yes
Description: The snmptrap() source automatically parses the traps into name-value pairs, so you can handle the content of the trap as a structured message. Consequently, you might not even need the ${MESSAGE} part of the log message. If set-message-macro() is set to no, AxoSyslog leaves the ${MESSAGE} part empty. If set-message-macro() is set to yes, AxoSyslog generates a regular log message from the trap.
7.26 - sun-streams: Collect messages on Sun Solaris
Solaris uses its STREAMS framework to send messages to the syslogd process. Solaris 2.5.1 and above uses an IPC called door in addition to STREAMS, to confirm the delivery of a message. The AxoSyslog application supports the IPC mechanism via the door() option (see below).
Note
The sun-streams() driver must be enabled when the syslog-ng application is compiled (see ./configure --help).
The sun-streams() driver has a single required argument specifying the STREAMS device to open, and the door() option. For the list of available optional parameters, see sun-streams() source options.
Note
Starting with version 3.7, the AxoSyslogsystem() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the AxoSyslog GitHub repository.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
follow-freq()
Type:
number
Default:
1
Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
log-fetch-limit()
Type:
number
Default:
100
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
100
Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
log-prefix() (DEPRECATED)
Type:
string
Default:
Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.
Note
This option is deprecated. Use program-override instead.
optional()
Type:
yes or no
Default:
Description: Instruct AxoSyslog to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
7.27 - syslog: Collect messages using the IETF-syslog protocol
The syslog() driver can receive messages from the network using the standard IETF-syslog protocol (as described in RFC5424-26). UDP, TCP, and TLS-encrypted TCP can all be used to transport the messages.
Note
The syslog() driver can also receive BSD-syslog-formatted messages (described in RFC 3164, see BSD-syslog or legacy-syslog messages) if they are sent using the IETF-syslog protocol.
In AxoSyslog versions 3.1 and earlier, the syslog() driver could handle only messages in the IETF-syslog (RFC 5424-26) format.
When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the AxoSyslog server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf() option of the source is increased. In this cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.
As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152 bytes.
7.27.1 - syslog() source options
The syslog() driver has the following options.
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
By default, the kernel chooses the receive socket for a specific UDP randomly based on the source IP/port of the sender. You can customize this algorithm using the Extended Berkeley Packet Filter (eBPF) plugin. The ebpf() option changes the kernel’s SO_REUSEPORT algorithm so that all messages are randomly placed into one of the UDP sockets. The decision which UDP socket buffer a datagram is placed is made for every datagram, and not once for every stream. This means that messages are perfectly load-balanced across your set of UDP sockets. While this resolves the imbalance between the sockets and results in perfect load balancing, you will lose ordering between messages from the same sender, which is the price to pay for increased throughput.
Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
threaded: The threaded flag enables multithreading for the source. For details on multithreading, see Multithreading and scaling.
Note
The syslog source uses multiple threads only if the source uses the tls or tcp transport protocols.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Replaces the ${HOST} part of the message with the parameter string.
interface()
Type:
string
Default:
None
Description: Bind to the specified interface instead of an IP address. Available in 3.19 and later.
ip() or localip()
Type:
string
Default:
0.0.0.0
Description: The IP address to bind to. By default, AxoSyslog listens on every available interface. Note that this is not the address where messages are accepted from.
If you specify a multicast bind address and use the udp transport, AxoSyslog automatically joins the necessary multicast group. TCP does not support multicasting.
ip-protocol()
Type:
number
Default:
4
Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).
Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).
For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.
Description: Specifies the Type-of-Service value of outgoing packets.
ip-ttl()
Type:
number
Default:
0
Description: Specifies the Time-To-Live value of outgoing packets.
keep-alive()
Type:
yes or no
Default:
yes
Description: Specifies whether connections to sources should be closed when AxoSyslog is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the AxoSyslog connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.
keep-hostname()
Type:
yes or no
Default:
no
Description: Enable or disable hostname rewriting.
If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.
If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.
Note
If the log message does not contain a hostname in its HOST field, AxoSyslog automatically adds a hostname to the message.
For messages received from the network, this hostname is the address of the host that sent the message (this means the address of the last hop if the message was transferred via a relay).
For messages received from the local host, AxoSyslog adds the name of the host.
This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
When relaying messages, enable this option on the AxoSyslog server and also on every relay, otherwise AxoSyslog will treat incoming messages as if they were sent by the last relay.
keep-timestamp()
Type:
yes or no
Default:
yes
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
listen-backlog()
Type:
integer
Default:
256
Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
100
Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.
If the max-connections() option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection. For optimal performance when receiving messages from AxoSyslog clients, make sure that the window size is larger than the flush-lines() option set in the destination of your clients.
Example: Initial window size of a connection
If log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
max-connections()
Type:
number
Default:
10
Description: Specifies the maximum number of simultaneous connections.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
port() or localport()
Type:
number
Default:
In case of TCP transport: 514
In case of UDP transport: 514
Description: The port number to bind to.
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
so-broadcast()
Type:
yes or no
Default:
no
Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.
so-keepalive()
Type:
yes or no
Default:
no
Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-rcvbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.
Warning
When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the AxoSyslog server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf() option of the source is increased. In this cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.
As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152 bytes.
so-reuseport()
Type:
yes or no
Default:
no
Description: Enables SO_REUSEPORT on systems that support it. When enabled, the kernel allows multiple UDP sockets to be bound to the same port, and the kernel load-balances incoming UDP datagrams to the sockets. The sockets are distributed based on the hash of (srcip, dstip, srcport, dstport), so the same listener should be receiving packets from the same endpoint. For example:
Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-sndbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
Description: Specifies the protocol used to receive messages from the source.
For detailed information about how AxoSyslog supports the proxied-tcp, the proxied-tls, and the proxied-tls-passthrough parameters, see Proxy Protocol support.
text-with-nuls: Allows embedded NUL characters in the message from a TCP source, that is, AxoSyslog will not delimiter the incoming messages on NUL characters, only on newline characters (contrary to tcp transport, which splits the incoming log on newline characters and NUL characters).
Note
The AxoSyslog application does not support embedded NUL characters everywhere, so it is recommended that you also use flags(no-multi-line) that causes NUL characters to be replaced by space.
Warning
When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the AxoSyslog server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf() option of the source is increased. In this cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.
As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152 bytes.
trim-large-messages()
Type:
yes
Default:
Use the global trim-large-messages() option, which defaults to no.
Description: Determines what AxoSyslog does with incoming log messages that are received using the IETF-syslog protocol using the syslog() driver, and are longer than the value of log-msg-size(). Other drivers ignore the trim-large-messages() option.
If set to no, AxoSyslog drops the incoming log message.
If set to yes, AxoSyslog trims the incoming log message to the size set in log-msg-size(), and adds the trimmed tag to the message. The rest of the message is dropped. You can use the tag to filter on such messages.
filter f_trimmed { tags("trimmed");};
If AxoSyslog trims a log message, it sends a debug-level log message to its internal() source.
As a result of trimming, a parser could fail to parse the trimmed message. For example, a trimmed JSON or XML message will not be valid JSON or XML.
Available in AxoSyslog version 3.21 and later.
Uses the value of the global option if not specified.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
use-dns()
Type:
yes, no, persist_only
Default:
yes
Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example, from /etc/hosts). The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
use-fqdn()
Type:
yes or no
Default:
no
Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.
Note
Set use-fqdn() to yes if you want to use the custom-domain() global option.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
7.28 - syslog-ng-otlp(): Receive logs from another node using OpenTelemetry
Available in AxoSyslog version 4.4 and later.
The syslog-ng-otlp() source and destination allows you to transfer the internal representation of log messages between AxoSyslog instances using the OpenTelemetry protocol. Unlike the traditional syslog-ng() drivers that rely on simple TCP connections, syslog-ng-otlp() leverages the OpenTelemetry protocol for efficient and reliable log message transmission.
The key benefits of the syslog-ng-otlp() drivers are:
scalability (via the workers() option),
built-in application layer acknowledgement,
support for Google service authentication (ADC or ALTS), and
improved load balancing capabilities.
To use it, configure a syslog-ng-otlp() destination on the sender node, and a syslog-ng-otlp() source on the receiver node, like this:
Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.
tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.
Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:
Description: Configures the maximal number of in-flight gRPC requests per worker. Setting this value in the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, workers() * concurrent-requests() should be greater than or equal to the number of clients, but this can increase the memory usage.
default-facility()
Type:
facility string
Default:
kern
Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.
default-level()
Type:
string
Default:
notice
Description: The default level value if the PRIORITY entry does not exist.
default-priority()
Type:
priority string
Default:
Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).
dns-cache()
Accepted values:
yes, no
Default:
no
Description: Enable or disable DNS cache usage.
ebpf()
Available in AxoSyslog version 4.2 and newer.
By default, the kernel chooses the receive socket for a specific UDP randomly based on the source IP/port of the sender. You can customize this algorithm using the Extended Berkeley Packet Filter (eBPF) plugin. The ebpf() option changes the kernel’s SO_REUSEPORT algorithm so that all messages are randomly placed into one of the UDP sockets. The decision which UDP socket buffer a datagram is placed is made for every datagram, and not once for every stream. This means that messages are perfectly load-balanced across your set of UDP sockets. While this resolves the imbalance between the sockets and results in perfect load balancing, you will lose ordering between messages from the same sender, which is the price to pay for increased throughput.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
host-override()
Type:
string
Default:
Description: Replaces the ${HOST} part of the message with the parameter string.
keep-hostname()
Type:
yes or no
Default:
no
Description: Enable or disable hostname rewriting.
If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.
If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.
Note
If the log message does not contain a hostname in its HOST field, AxoSyslog automatically adds a hostname to the message.
For messages received from the network, this hostname is the address of the host that sent the message (this means the address of the last hop if the message was transferred via a relay).
For messages received from the local host, AxoSyslog adds the name of the host.
This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
When relaying messages, enable this option on the AxoSyslog server and also on every relay, otherwise AxoSyslog will treat incoming messages as if they were sent by the last relay.
keep-timestamp()
Type:
yes or no
Default:
yes
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
log-fetch-limit()
Type:
number
Default:
100
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
10000
Description: The size of the initial window, this value is used during flow control. Make sure that log-iw-size() is larger than the value of log-fetch-limit().
log-prefix() (DEPRECATED)
Type:
string
Default:
Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.
Note
This option is deprecated. Use program-override instead.
normalize-hostnames()
Accepted values:
yes, no
Default:
no
Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.
Note
This setting applies only to hostnames resolved from DNS. It has no effect if the keep-hostname() option is enabled, and the message contains a hostname.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
port()
Type:
integer
Default:
Description: The port number to bind to.
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
use-dns()
Type:
yes, no, persist_only
Default:
yes
Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example, from /etc/hosts). The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
use-fqdn()
Type:
yes or no
Default:
no
Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.
Note
Set use-fqdn() to yes if you want to use the custom-domain() global option.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to process the messages of the source. Increasing the number of worker threads can drastically improve the performance of the source.
7.29 - system: Collect the system-specific log messages of a platform
Starting with version 3.2, AxoSyslog can automatically collect the system-specific log messages of the host on a number of platforms using the system() driver. If the system() driver is included in the AxoSyslog configuration file, AxoSyslog automatically adds the following sources to the AxoSyslog configuration.
Note
AxoSyslog versions 3.2-3.3 used an external script to generate the system() source, but this was problematic in certain situations, for example, when the host used a strict AppArmor profile. Therefore, the system() source is now generated internally in AxoSyslog.
The system() driver is also used in the default configuration file of AxoSyslog. For details on the default configuration file, see Example: The default configuration file of [%=General.OSE%]. Starting with AxoSyslog version 3.6, you can use the system-expand command-line utility (which is a shell script, located in the modules/system-source/ directory) to display the configuration that the system() source will use.
Warning
If AxoSyslog does not recognize the platform it is installed on, it does not add any sources.
Starting with version 3.6, AxoSyslog parses messages complying with the Splunk Common Information Model (CIM) and marked with @cim as JSON messages (for example, the ulogd from the netfilter project can emit such messages). That way, you can forward such messages without losing any information to CIM-aware applications (for example, Splunk).
Note that on Linux, the so-rcvbuf() option of the system() source is automatically set to 8192.
If the host is running under systemd, AxoSyslog reads both syslog and kernel messages directly from the systemd journal file using the systemd-journal() source. In this case, AxoSyslog doesn't read from `/dev/log` nor `/proc/kmsg`.
If the kernel of the host is version 3.5 or newer, and /dev/kmsg is seekable, AxoSyslog will use that instead of /proc/kmsg, using the multi-line-mode(indented), keep-timestamp(no), and the format(linux-kmsg) options.
If AxoSyslog is running in a jail or a Linux Container (LXC), it will not read from the /dev/kmsg or /proc/kmsg files.
Starting with version 3.7, the AxoSyslogsystem() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the AxoSyslog GitHub repository.
NetBSD
unix-dgram("/var/run/log");
Note
Starting with version 3.7, the AxoSyslogsystem() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the AxoSyslog GitHub repository.
Solaris 8
sun-streams("/dev/log");
Note
Starting with version 3.7, the AxoSyslogsystem() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the AxoSyslog GitHub repository.
Starting with version 3.7, the AxoSyslogsystem() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the AxoSyslog GitHub repository.
Starting with version 3.7, the AxoSyslogsystem() driver automatically extracts the msgid from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the extract-solaris-msgid() parser. You can find the exact source of this parser in the AxoSyslog GitHub repository.
7.29.1 - system() source options
The system() driver has the following options:
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
7.30 - systemd-journal: Collect messages from the systemd-journal system log storage
The systemd-journal() source is used on various Linux distributions, such as RHEL (from RHEL7) and CentOS. The systemd-journal() source driver can read the structured name-value format of the journald system service, making it easier to reach the custom fields in the message. By default, AxoSyslog adds the .journald. prefix to the name of every parsed value. For a list and description of name-value pairs that journald provides, see the documentation of journald for your platform (for example, man systemd.journal-fields).
The systemd-journal() source driver is designed to read only local messages through the systemd-journal API. It is not possible to set the location of the journal files, or the directories.
Note
The log-msg-size() option is not applicable for this source. Use the max-field-size() option instead.
Note
This source will not handle the following cases:
Corrupted journal file
Incorrect journal configuration
Any other journald-related bugs
Note
If you are using RHEL-7, the default source in the configuration is systemd-journal() instead of unix-dgram("/dev/log") and file("/proc/kmsg"). If you are using unix-dgram("/dev/log") or unix-stream("/dev/log") in your configuration as a source, AxoSyslog will revert to using systemd-journal() instead.
Warning
Only one systemd-journal() source can be configured in the configuration file. If there is more than one systemd-journal() source configured, AxoSyslog will not start.
Declaration:
systemd-journal(options);
If you want to use multiple systemd-journal() sources in your configuration, the sources must use unique systemd namespaces. For details, see the namespace() option.
Example: Send all fields through syslog protocol
To send all fields through the syslog protocol, enter the prefix in the following format: “.SDATA.<name>”.
The journal contains credential information about the process that sent the log message. The AxoSyslog application makes this information available in the following macros:
Journald fields as macros
Journald field
AxoSyslog predefined macro
MESSAGE
$MESSAGE
_HOSTNAME
$HOST
_PID
$PID
_COMM or SYSLOG_IDENTIFIER
$PROGRAM If both _COMM and SYSLOG_IDENTIFIER exists, AxoSyslog uses SYSLOG_IDENTIFIER
SYSLOG_FACILITY
$FACILITY_NUM
PRIORITY
$LEVEL_NUM
7.30.1 - systemd-journal() source options
The systemd-journal() driver has the following options:
default-facility()
Type:
facility string
Default:
local0
Description: The default facility value if the SYSLOG_FACILITY entry does not exist.
default-level()
Type:
string
Default:
notice
Description: The default level value if the PRIORITY entry does not exist.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Replaces the ${HOST} part of the message with the parameter string.
keep-hostname()
Type:
yes or no
Default:
no
Description: Enable or disable hostname rewriting.
If enabled (keep-hostname(yes)), AxoSyslog will retain the hostname information read from the systemd journal messages.
If disabled (keep-hostname(no)), AxoSyslog will use the hostname that has been set up for the operating system instance that AxoSyslog is running on. To query or set this value, use the hostnamectl command.
This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
match-boot()
Type:
yes, no
Default:
no
Available in AxoSyslog 4.1 and later.
Description: If set to yes, AxoSyslog fetches only journal messages that relate to the current boot, and to ignores messages generated in previous boots.
matches()
Type:
string
Default:
Available in AxoSyslog 4.1 and later.
Description: Specifies one or more filters to apply on the journal fields, similarly how you can use journalctl. For example:
matches("_COMM"=> "systemd")
max-field-size()
Type:
number (characters)
Default:
65536
Description: The maximum length of a field’s value.
If you do not specify the namespace() option in your configuration, or if you specify an empty string, the systemd-journal() source reads and displays log data from all namespaces.
If you specify the namespace() option as namespace("*"), the systemd-journal() source reads and displays log data from all namespaces, interleaved.
If namespace(<specific-namespace>) is specified, the systemd-journal() source only reads and displays log data from the specified namespace.
If the namespace identifier is prefixed with "+" when you specify your namespace() option, the systemd-journal() source only reads and displays log data from the specified namespace and the default namespace, interleaved.
Starting with AxoSyslog 4.4, you can use multiple systemd-journal() sources in your configuration. In this case, each source must use unique systemd namespaces.
Syntax:namespace(string)
Example: configuration examples for using the namespace() option
The following configuration example uses the default value for the namespace() option:
Namespace support was introduced to the Journalctl command line tool in Systemd version 2.45. The AxoSyslog application supports the namespace() option from version 3.29. For further information about namespaces on the Systemd side, see Journal Namespaces.
prefix()
Type:
string
Default:
.journald.
Description: If this option is set, every non-built-in mapped names get a prefix (for example: ".SDATA.journald."). By default, AxoSyslog adds the .journald. prefix to every value.
read-old-records()
Type:
yes
Default:
yes
Description: If set to yes, AxoSyslog will start reading the records from the beginning of the journal, if the journal has not been read yet. If set to no, AxoSyslog will read only the new records. If the source has a state in the persist file, this option will have no effect.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
use-fqdn()
Type:
yes or no
Default:
no
Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.
Note
Set use-fqdn() to yes if you want to use the custom-domain() global option.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
7.31 - systemd-syslog: Collect systemd messages using a socket
On platforms running systemd, the systemd-syslog() driver reads the log messages of systemd using the /run/systemd/journal/syslog socket. Note the following points about this driver:
If possible, use the more reliable systemd-journal() driver instead.
The socket activation of systemd is buggy, causing some log messages to get lost during system startup.
If AxoSyslog is running in a jail or a Linux Container (LXC), it will not read from the /dev/kmsg or /proc/kmsg files.
The systemd-syslog() driver has the following options:
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
The tcp(), tcp6(), udp(), udp6() drivers can receive syslog messages conforming to RFC3164 from the network using the TCP and UDP networking protocols. The tcp6() and udp6() drivers use the IPv6 network protocol, while tcp() and udp() use IPv4.
7.32.1.1 - Change an old source driver to the network() driver
To replace your existing tcp(), tcp6(), udp(), udp6() sources with a network() source, complete the following steps.
Replace the driver with network. For example, replace udp( with network(
Set the transport protocol.
If you used TLS-encryption, add the transport("tls") option, then continue with the next step.
If you used the tcp or tcp6 driver, add the transport("tcp") option.
If you used the udp or udp driver, add the transport("udp") option.
If you use IPv6 (that is, the udp6 or tcp6 driver), add the ip-protocol(6) option.
If you did not specify the port used in the old driver, check network() source options and verify that your clients send the messages to the default port of the transport protocol you use. Otherwise, set the appropriate port number in your source using the port() option.
All other options are identical. Test your configuration with the syslog-ng --syntax-only command.
The following configuration shows a simple tcp source.
7.33 - unix-stream, unix-dgram: Collect messages from UNIX domain sockets
The unix-stream() and unix-dgram() drivers open an AF_UNIX socket and start listening on it for messages. The unix-stream() driver is primarily used on Linux and uses SOCK_STREAM semantics (connection oriented, no messages are lost), while unix-dgram() is used on BSDs and uses SOCK_DGRAM semantics: this may result in lost local messages if the system is overloaded.
To avoid denial of service attacks when using connection-oriented protocols, the number of simultaneously accepted connections should be limited. This can be achieved using the max-connections() parameter. The default value of this parameter is quite strict, you might have to increase it on a busy system.
Both unix-stream and unix-dgram have a single required argument that specifies the filename of the socket to create. For the list of available optional parameters, see unix-stream() and unix-dgram() source options
syslogd on Linux originally used SOCK_STREAM sockets, but some distributions switched to SOCK_DGRAM around 1999 to fix a possible DoS problem. On Linux you can choose to use whichever driver you like as syslog clients automatically detect the socket type being used.
Example: Using the unix-stream() and unix-dgram() drivers
Starting with AxoSyslog 3.6, the unix-stream() and unix-dgram() sources automatically extract the available UNIX credentials and other metainformation from the received log messages. The AxoSyslog application can extract the following information on Linux and FreeBSD platforms (examples show the value of the macro for the su - myuser command). Similar information is available for the systemd-journal source.
UNIX credentials available via UNIX domain sockets
Macro
Description
${.unix.cmdline}
The name (without the path) and command-line options of the executable belonging to the PID that sent the message. For example, su - myuser
${.unix.exe}
The path of the executable belonging to the PID that sent the message. For example, /usr/bin/su
${.unix.gid}
The group ID (GID) corresponding to the UID of the application that sent the log message. Note that this is the ID number of the group, not its human-readable name. For example, 0
${.unix.pid}
The process ID (PID) of the application that sent the log message. For example, 774.
Note that on every UNIX platforms, if the `system()` source uses sockets, it will overwrite the PID macro with the value of `${.unix.pid}`, if it is available.
${.unix.uid}
The user ID (UID) of the application that sent the log message. Note that this is the ID number of the user, not its human-readable name. For example, 0
7.33.2 - unix-stream() and unix-dgram() source options
These two drivers behave similarly: they open an AF_UNIX socket and start listening on it for messages. The following options can be specified for these drivers:
create-dirs()
Type:
yes or no
Default:
no
Description: Enable creating non-existing directories when creating files or socket files.
encoding()
Type:
string
Default:
Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
group()
Type:
string
Default:
root
Description: Set the gid of the socket.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Replaces the ${HOST} part of the message with the parameter string.
keep-alive()
Type:
yes or no
Default:
yes
Description: Selects whether to keep connections open when syslog-ng is restarted, cannot be used with unix-dgram().
keep-timestamp()
Type:
yes or no
Default:
yes
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
listen-backlog()
Type:
integer
Default:
256
Description: Available only for stream based transports (unix-stream, tcp, tls). In TCP, connections are treated incomplete until the three-way handshake is completed between the server and the client. Incomplete connection requests wait on the TCP port for the listener to accept the request. The listen-backlog() option sets the maximum number of incomplete connection requests. For example:
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
100
Description: The size of the initial window, this value is used during flow-control. Its value cannot be lower than 100, unless the dynamic-window-size() option is enabled. For details on flow-control, see Managing incoming and outgoing messages with flow-control.
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
log-prefix() (DEPRECATED)
Type:
string
Default:
Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.
Note
This option is deprecated. Use program-override instead.
max-connections()
Type:
number (simultaneous connections)
Default:
256
Description: Limits the number of simultaneously open connections. Cannot be used with unix-dgram().
optional()
Type:
yes or no
Default:
Description: Instruct AxoSyslog to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.
owner()
Type:
string
Default:
root
Description: Set the uid of the socket.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
perm()
Type:
number (octal notation)
Default:
0666
Description: Set the permission mask. For octal numbers prefix the number with ‘0’, for example: use 0755 for rwxr-xr-x.
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
so-keepalive()
Type:
yes or no
Default:
no
Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-rcvbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.
Warning
When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the AxoSyslog server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf() option of the source is increased. In this cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.
As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152 bytes.
so-reuseport()
Type:
yes or no
Default:
no
Description: Enables SO_REUSEPORT on systems that support it. When enabled, the kernel allows multiple UDP sockets to be bound to the same port, and the kernel load-balances incoming UDP datagrams to the sockets. The sockets are distributed based on the hash of (srcip, dstip, srcport, dstport), so the same listener should be receiving packets from the same endpoint. For example:
Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
7.34 - stdin: Collect messages from the standard input stream
The stdin() driver collects messages from the standard input stream. When the standard input stream is closed, AxoSyslog stops and stdin() inherits all options from the file() source, including multi-line options, or flags(no-parse).
The stdin() driver causes AxoSyslog to exit once it hits end-of-file (EOF).
The following code snippet is an example of how the stdin() driver is used to collect a test message:
$ echo"this is a test message"| ./syslog-ng -Fe --no-caps
[2017-11-14T13:47:16.757938] syslog-ng starting up;version='3.12.1'[2017-11-14T13:47:16.758195] syslog-ng shutting down;version='3.12.1' Nov 14 13:47:16 testserver this is a test message
7.34.1 - stdin() source options
The stdin() driver has the following options:
default-facility()
Type:
facility string
Default:
kern
Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.
default-priority()
Type:
priority string
Default:
Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).
encoding()
Type:
string
Default:
Description: Specifies the character set (encoding, for example, UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command. For details on how encoding affects the size of the message, see Message size and encoding.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
follow-freq()
Type:
number
Default:
1
Description: Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, AxoSyslog will not attempt to use poll() on the file, but checks whether the file changed every time the follow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example, 1.5) can be used as well.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
log-fetch-limit()
Type:
number
Default:
100
Description: The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.
log-iw-size()
Type:
number
Default:
10000
Description: The size of the initial window, this value is used during flow control. Make sure that log-iw-size() is larger than the value of log-fetch-limit().
log-msg-size()
Type:
number (bytes)
Default:
Use the global log-msg-size() option, which defaults to 65536 (64 KiB).
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, log-msg-size() does not need to be set higher than 10 MiB.
Uses the value of the global option if not specified.
log-prefix() (DEPRECATED)
Type:
string
Default:
Description: A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux.
Note
This option is deprecated. Use program-override instead.
multi-line-garbage()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.
When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.
To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.
Warning
If the multi-line-garbage() option is set, AxoSyslog discards lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix().
Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:
indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.
prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.
prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.
The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.
smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.
smart mode is available in AxoSyslog version 4.2 and newer.
The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
multi-line-prefix()
Type:
regular expression starting with the ^ character
Default:
empty string
Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
Example: Processing Tomcat logs
The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.
2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.net.BindException: Address already in use null:8080
at org.apache.catalina.connector.Connector.start(Connector.java:1138) at org.apache.catalina.core.StandardService.start(StandardService.java:531) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177) 2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
INFO: Server startup in 1206 ms
2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:
Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.
multi-line-suffix()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.
To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.
pad-size()
Type:
number
Default:
0
Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. The AxoSyslog application will pad reads from the associated device to the number of bytes set in pad-size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() was given and the incoming message does not fit into pad-size(), AxoSyslog will not read anymore from this pipe and displays the following error message:
Padding was set, and couldn't read enough bytes
program-override()
Type:
string
Default:
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
tags()
Type:
string
Default:
Description: Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example, tags("dmz", "router"). This option is available only in version 3.1 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
Description: The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
8 - destination: Forward, send, and store log messages
A destination is where a log message is sent if the filtering rules match. Similarly to sources, destinations consist of one or more drivers, each defining where and how messages are sent.
Note
If no drivers are defined for a destination, all messages sent to the destination are discarded. This is equivalent to omitting the destination from the log statement.
To define a destination, add a destination statement to the syslog-ng.conf configuration file using the following syntax:
Do not define the same drivers with the same parameters more than once, because it will cause problems. For example, do not open the same file in multiple destinations.
Do not use the same destination in different log paths, because it can cause problems with most destination types. Instead, use filters and log paths to avoid such situations.
Sources and destinations are initialized only when they are used in a log statement. For example, AxoSyslog starts listening on a port or starts polling a file only if the source is used in a log statement. For details on creating log statements, see log: Filter and route log messages using log paths, flags, and filters.
The following table lists the destination drivers available in AxoSyslog. If these destinations do not satisfy your needs, you can extend AxoSyslog and write your own destination, for example, in C, Java, or Python. For details, see Write your own custom destination in Java or Python.
The following destination driver groups are available in AxoSyslog:
8.1 - amqp: Publish messages using AMQP
The amqp() driver publishes messages using the AMQP (Advanced Message Queuing Protocol). AxoSyslog supports AMQP versions 0.9.1 and 1.0. The AxoSyslog amqp() driver supports persistence, and every available exchange types.
The name-value pairs selected with the value-pairs() option will be sent as AMQP headers, while the body of the AMQP message is empty by default (but you can add custom content using the body() option). Publishing the name-value pairs as headers makes it possible to use the Headers exchange-type and subscribe only to interesting log streams. This solution is more flexible than using the routing-key() option.
The amqp() driver publishes messages using the AMQP (Advanced Message Queuing Protocol).
The amqp() destination has the following options:
auth-method()
Accepted values:
plain
Default:
plain
Description: The amqp() driver supports the following types of authentication:
plain: Authentication happens using username and password. This is the default.
external: Authentication happens using an out-of-band mechanism, for example, x509 certificate peer verification, client IP address range, or similar. For more information, see the RabbitMQ documentation.
batch-bytes()
Accepted values:
number [bytes]
Default:
none
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
body()
Type:
string
Default:
empty string
Description: The body of the AMQP message. You can also use macros and templates.
ca-file()
Type:
string
Default:
N/A
Description: Name of a file, that contains the trusted CA certificate in PEM format. For example: ca-file("/home/certs/syslog-ng/tls/cacert.pem"). The AxoSyslog application uses this CA certificate to validate the certificate of the peer.
An alternative way to specify this option is to put into a tls() block and specify it there, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-file(), cert-file(), key-file(), and peer-verify()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
cert-file()
Accepted values:
Filename
Default:
none
Description: Name of a file, that contains an X.509 certificate (or a certificate chain) in PEM format, suitable as a TLS certificate, matching the private key set in the key-file() option. The AxoSyslog application uses this certificate to authenticate the AxoSyslog client on the destination server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put into a tls() block and specify it there, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-file(), cert-file(), key-file(), and peer-verify()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: The name of the AMQP exchange where AxoSyslog sends the message. Exchanges take a message and route it into zero or more queues.
exchange-declare()
Type:
yes
Default:
no
Description: By default, AxoSyslog does not create non-existing exchanges. Use the exchange-declare(yes) option to automatically create exchanges.
exchange-type()
Type:
direct
Default:
fanout
Description: The type of the AMQP exchange.
frame-size()
Type:
integer
Default:
Description: Sets maximal frame size (the frame-max option described in the AMQP Reference Guide.
heartbeat()
Type:
number [seconds]
Default:
0 (disabled)
Description: If enabled, the AxoSyslog amqp destination sends heartbeat messages to the server periodically. During negotiation, both the amqp server and the client provide a heartbeat parameter, and the smaller is chosen for heartbeat interval. For example:
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The hostname or IP address of the AMQP server.
key-file()
Accepted values:
Filename
Default:
none
Description: The name of a file that contains an unencrypted private key in PEM format, suitable as a TLS key. If properly configured, the AxoSyslog application uses this private key and the matching certificate (set in the cert-file() option) to authenticate the AxoSyslog client on the destination server.
max-channel()
Type:
integer
Default:
Description: Sets maximal number of channels (the channel-max option described in the AMQP Reference Guide.
An alternative way to specify this option is to put into a tls() block and specify it there, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-file(), cert-file(), key-file(), and peer-verify()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
password()
Type:
string
Default:
n/a
Description: The password used to authenticate on the AMQP server.
peer-verify()
Accepted values:
`yes
Default:
yes
Description: Verification method of the peer. The following table summarizes the possible options and their results depending on the certificate of the peer.
The remote peer has:
no certificate
invalid certificate
valid certificate
Local peer-verify() setting
no (optional-untrusted)
TLS-encryption
TLS-encryption
TLS-encryption
yes (required-trusted)
rejected connection
rejected connection
TLS-encryption
For untrusted certificates only the existence of the certificate is checked, but it does not have to be valid — AxoSyslog accepts the certificate even if it is expired, signed by an unknown CA, or its CN and the name of the machine mismatches.
Warning
When validating a certificate, the entire certificate chain must be valid, including the CA certificate. If any certificate of the chain is invalid, AxoSyslog will reject the connection.
An alternative way to specify this option is to put into a tls() block and specify it there, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-file(), cert-file(), key-file(), and peer-verify()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
persistent()
Type:
yes
Default:
yes
Description: If this option is enabled, the AMQP server or broker will store the messages on its hard disk. That way, the messages will be retained if the AMQP server is restarted, if the message queue is set to be durable on the AMQP server.
port()
Type:
number
Default:
5672
Description: The port number of the AMQP server.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
routing-key()
Type:
string
Default:
empty string
Description: Specifies a routing key for the exchange. The routing key selects certain messages published to an exchange to be routed to the bound queue. In other words, the routing key acts like a filter. The routing key can include macros and templates.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
username()
Type:
string
Default:
empty string
Description: The username used to authenticate on the AMQP server.
value-pairs()
Type:
parameter list of the value-pairs() option
Default:
scope("selected-macros" "nv-pairs")
Description: The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Structuring macros, metadata, and other value-pairs.
Note
Empty keys are not logged.
vhost()
Type:
string
Default:
/
Description: The name of the AMQP virtual host to send the messages to.
For authentication, the destination uses GoogleDefaultCredentials, which covers everything listed as ADC. In a production environment, use a service account and Workload Identity.
By default, the messages are sent with one worker, one message per batch, and without compression.
Options
The bigquery() destination has the following options.
batch-bytes()
Accepted values:
number [bytes]
Default:
none
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
By default, the batch-bytes() option of the bigquery() destination is 10 MB. This is an upper limit for the bigquery() destination. Note that due to a framework limitation, the batch might be at most 1 message larger than the set limit.
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
channel-args()
Type:
See description
Default:
-
Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:
Description: Enables compression in gRPC requests. Although gRPC supports various compression methods, currently only deflate is supported (which is basically the same as gzip).
dataset()
Type:
string
Default:
-
Description: The name of the dataset where AxoSyslog sends the data.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The maximum number of gRPC pings that can be sent when there is no data/header frame to be sent. AxoSyslog won’t send any pings after this limit. Set it to 0 disable this restriction and keep sending pings.
time()
Type:
number [milliseconds]
Default:
Description: The period (in milliseconds) after which AxoSyslog sends a gRPC keepalive ping.
timeout()
Type:
number [milliseconds]
Default:
10
Description: The time (in milliseconds) AxoSyslog waits for an acknowledgement.
local-time-zone()
Type:
name of the timezone, or the timezone offset
Default:
The local timezone.
Description: Sets the timezone used when expanding filename and tablename templates.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
on-error()
Type:
One of: drop-message, drop-property, fallback-to-string, silently-drop-message, silently-drop-property, silently-fallback-to-string
Default:
Use the global setting (which defaults to drop-message)
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
project()
Type:
string
Default:
-
Description: The ID of the Google Cloud project where AxoSyslog sends the data.
protobuf-schema()
Type:
See the description
Default:
-
Description: Sets the schema of the BigQuery table from a protobuf schema file.
Alternatively, you can set the schema with the schema() option.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
schema()
Type:
See the description
Default:
-
Description: Sets the schema of the BigQuery table. On the left side of the arrow, set the name of the column and its type. On the right side, set any AxoSyslog template or macro, which gets evaluated on each log that is routed to the bigquery() destination. The available column types are: STRING, BYTES, INTEGER, FLOAT, BOOLEAN, TIMESTAMP, DATE, TIME, DATETIME, JSON, NUMERIC, BIGNUMERIC, GEOGRAPHY, RECORD, INTERVAL. For example:
Alternatively, you can set the schema with the protobuf-schema() option.
send-time-zone()
Accepted values:
name of the timezone, or the timezone offset
Default:
local timezone
Description: Specifies the time zone associated with the messages sent by syslog-ng, if not specified otherwise in the message or in the destination driver. For details, see Timezones and daylight saving.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
table()
Type:
string
Default:
-
Description: The name of the Google BigQuery table where AxoSyslog sends the data.
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
url()
Type:
string
Default:
bigquerystorage.googleapis.com
Description: The URL of the Google BigQuery where the logs are sent.
worker-partition-key()
Type:
template
Default:
Description: The worker-partition-key() option specifies a template: messages that expand the template to the same value are mapped to the same partition. When batching is enabled and multiple workers are configured, it’s important to add only those messages to a batch which generate identical URLs. To achieve this, set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed.
For example, you can partition messages based on the destination host:
worker-partition-key("$HOST")
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
The following example uses the name of the application sending the log message as the plugin name, and the value of the ${SEQNUM} macro as the value of the metric sent to collectd.
To use the collectd() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The collectd() driver is actually a reusable configuration snippet configured to send log messages using the unix-stream() driver. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
8.3.1 - collectd() destination options
The collectd() destination has the following options. The plugin() and type() options are required options. You can also set other options of the underlying unix-stream() driver (for example, socket buffer size).
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The hostname that is passed to collectd. By default, AxoSyslog uses the host from the log message as the hostname.
type("gauge"),
interval()
Type:
integer
Default:
60
Description: The interval in which the data is collected.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
keep-alive()
Type:
yes or no
Default:
yes
Description: Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.
plugin()
Type:
string
Default:
Description: The name of the plugin that submits the data to collectd. For example:
plugin("${PROGRAM}"),
plugin-instance()
Type:
string
Default:
Description: The name of the plugin-instance that submits the data to collectd.
Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.
so-keepalive()
Type:
yes or no
Default:
no
Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-rcvbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.
so-sndbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
type()
Type:
string or template
Default:
Description: Identifies the type and number of values passed to collectd. For details, see the types.db manual page. For example:
type("gauge"),
type-instance()
Type:
string
Default:
Description: For example:
type-instance("seqnum"),
values()
Type:
string, macro,or template
Default:
U
Description: Colon-separated list of the values to send to collectd. For example:
values("${SEQNUM}"),
8.4 - discord: Send alerts and notifications to Discord
The discord() destination driver sends messages to Discord using Discord Webhook. For the list of available optional parameters, see Discord destination options.
By default the message sending is throttled to 5 message/sec, see Discord: Rate Limits. To change this, use the throttle() option.
To use this destination, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The discord() driver is actually a reusable configuration snippet configured to send log messages using the http() driver. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
The discord() destination of AxoSyslog can directly post log messages to web services using the HTTP protocol. The discord() destination has the following options.
avatar-url()
Type:
URL
Default:
N/A
Description: A hyperlink for icon of the author to be displayed in Discord. For details, see the avatar_url option in the Discord documentation.
batch-bytes()
Accepted values:
number [bytes]
Default:
none
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Name of a file, that contains an X.509 certificate (or a certificate chain) in PEM format, suitable as a TLS certificate, matching the private key set in the key-file() option. The AxoSyslog application uses this certificate to authenticate the AxoSyslog client on the destination server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Depends on the OpenSSL version that AxoSyslog uses
Description: Specifies the cipher, hash, and key-exchange algorithms used for the encryption, for example, ECDHE-ECDSA-AES256-SHA384. The list of available algorithms depends on the version of OpenSSL used to compile AxoSyslog. To specify multiple ciphers, separate the cipher names with a colon, and enclose the list between double-quotes, for example:
For a list of available algorithms, execute the openssl ciphers -v command. The first column of the output contains the name of the algorithms to use in the cipher-suite() option, the second column specifies which encryption protocol uses the algorithm (for example, TLSv1.2). That way, the cipher-suite() also determines the encryption protocol used in the connection: to disable SSLv3, use an algorithm that is available only in TLSv1.2, and that both the client and the server supports. You can also specify the encryption protocols using ssl-options().
You can also use the following command to automatically list only ciphers permitted in a specific encryption protocol, for example, TLSv1.2:
echo"cipher-suite(\"$(openssl ciphers -v | grep TLSv1.2 | awk '{print $1}'| xargs echo -n | sed 's/ /:/g'| sed -e 's/:$//')\")"
Note that starting with version 3.10, when AxoSyslog receives TLS-encrypted connections, the order of ciphers set on the AxoSyslog server takes precedence over the client settings.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The number of messages that the output queue can store.
key-file()
Accepted values:
Filename
Default:
none
Description: The name of a file that contains an unencrypted private key in PEM format, suitable as a TLS key. If properly configured, the AxoSyslog application uses this private key and the matching certificate (set in the cert-file() option) to authenticate the AxoSyslog client on the destination server.
The http() destination supports only unencrypted key files (that is, the private key cannot be password-protected).
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Removes every character above the set limit. For details, see the content option in the Discord documentation.
peer-verify()
Accepted values:
`yes
Default:
yes
Description: Verification method of the peer. The following table summarizes the possible options and their results depending on the certificate of the peer.
The remote peer has:
no certificate
invalid certificate
valid certificate
Local peer-verify() setting
no (optional-untrusted)
TLS-encryption
TLS-encryption
TLS-encryption
yes (required-trusted)
rejected connection
rejected connection
TLS-encryption
For untrusted certificates only the existence of the certificate is checked, but it does not have to be valid — AxoSyslog accepts the certificate even if it is expired, signed by an unknown CA, or its CN and the name of the machine mismatches.
Warning
When validating a certificate, the entire certificate chain must be valid, including the CA certificate. If any certificate of the chain is invalid, AxoSyslog will reject the connection.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
proxy()
Type:
The proxy server address, in `proxy("PROXY_IP:PORT")` format.
For example, `proxy("http://myproxy:3128")`
Default:
None
Description:
You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.
Alternatively, you can leave the HTTP as-is, in which case the driver leaves the default http_proxy and https_proxy environment variables unmodified.
Note
Configuring the <code>proxy()</code> option overwrites the default <code>http_proxy</code> and <code>https_proxy</code> environment variables.
Example: the proxy() option in configuration
The following example illustrates including the proxy() option in your configuration.
Description: Specifies what AxoSyslog does with the log message, based on the response code received from the HTTP server. If the server returns a status code beginning with 2 (for example, 200), AxoSyslog assumes the message was successfully sent. Otherwise, the action listed in the following table is applied. For status codes not listed in the following table, if the status code begins with 2 (for example, 299), AxoSyslog assumes the message was successfully sent. For other status codes, AxoSyslog disconnects. The following actions are possible:
disconnect: Keep trying to resend the message indefinitely.
drop: Drop the message without trying to resend it.
retry: Retry sending the message for a maximum of retries() times (3 by default).
success: Assume the message was successfully sent.
|------+-----------------------------------+------------|| code | explanation | action ||------+-----------------------------------+------------||100|"Continue"| disconnect ||101|"Switching Protocols"| disconnect ||102|"Processing"| retry ||103|"Early Hints"| retry ||200|"OK"| success ||201|"Created"| success ||202|"Accepted"| success ||203|"Non-Authoritative Information"| success ||204|"No Content"| success ||205|"Reset Content"| success ||206|"Partial Content"| success ||300|"Multiple Choices"| disconnect ||301|"Moved Permanently"| disconnect ||302|"Found"| disconnect ||303|"See Other"| disconnect ||304|"Not Modified"| retry ||307|"Temporary Redirect"| disconnect ||308|"Permanent Redirect"| disconnect ||400|"Bad Request"| disconnect ||401|"Unauthorized"| disconnect ||402|"Payment Required"| disconnect ||403|"Forbidden"| disconnect ||404|"Not Found"| disconnect ||405|"Method Not Allowed"| disconnect ||406|"Not Acceptable"| disconnect ||407|"Proxy Authentication Required"| disconnect ||408|"Request Timeout"| disconnect ||409|"Conflict"| disconnect ||410|"Gone"| drop ||411|"Length Required"| disconnect ||412|"Precondition Failed"| disconnect ||413|"Payload Too Large"| disconnect ||414|"URI Too Long"| disconnect ||415|"Unsupported Media Type"| disconnect ||416|"Range Not Satisfiable"| drop ||417|"Expectation Failed"| disconnect ||418|"I'm a teapot"| disconnect ||421|"Misdirected Request"| disconnect ||422|"Unprocessable Entity"| drop ||423|"Locked"| disconnect ||424|"Failed Dependency"| drop ||425|"Too Early"| drop ||426|"Upgrade Required"| disconnect ||428|"Precondition Required"| retry ||429|"Too Many Requests"| disconnect ||431|"Request Header Fields Too Large"| disconnect ||451|"Unavailable For Legal Reasons"| drop ||500|"Internal Server Error"| disconnect ||501|"Not Implemented"| disconnect ||502|"Bad Gateway"| disconnect ||503|"Service Unavailable"| disconnect ||504|"Gateway Timeout"| retry ||505|"HTTP Version Not Supported"| disconnect ||506|"Variant Also Negotiates"| disconnect ||507|"Insufficient Storage"| disconnect ||508|"Loop Detected"| drop ||510|"Not Extended"| disconnect ||511|"Network Authentication Required"| disconnect ||------+-----------------------------------+------------|
To customize the action to take for a particular response code, use the following format: response-action(<response-code> => <action>. To customize multiple response code-action pairs, separate them with a comma, for example:
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
To handle HTTP error responses, if the HTTP server returns 5xx codes, AxoSyslog will attempt to resend messages until the number of attempts reaches retries. If the HTTP server returns 4xx codes, AxoSyslog will drop the messages.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
Warning
Hazard of data loss! Make sure to include a fallback value, as if the template gets resolved to an empty string, Discord rejects the message.
throttle()
Type:
number
Default:
5
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
Description: The value (in seconds) to wait for an operation to complete, and attempt to reconnect the server if exceeded. By default, the timeout value is 0, meaning that there is no timeout. Available in version 3.11 and later.
tts()
Type:
`true
Default:
false
Description: Enables TTS (Text-To-Speech) mode. For more information, see the tts option in the Discord documentation.
url()
Type:
URL
Default:
N/A
Description: The webhook URL of the Discord server/channel. For more information, see Discord: Intro to Webhooks.
user-agent()
Type:
string
Default:
syslog-ng [version]'/libcurl[version]`
Description: The value of the USER-AGENT header in the messages sent to the server.
username()
Type:
string
Default:
N/A
Description: Overrides the default username of the webhook. For details, see the username option in the Discord documentation.
use-system-cert-store()
Type:
`yes
Default:
no
Description: Use the certificate store of the system for verifying HTTPS certificates. For details, see the curl documentation.
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1", "site2", "site3")), set the workers() option to 3 or more.
8.5 - elasticsearch2: DEPRECATED - Send messages directly to Elasticsearch version 2.0 or higher
Starting with version 3.7 of AxoSyslog can directly send log messages to Elasticsearch, allowing you to search and analyze your data in real time, and visualize it with Kibana.
Note the following limitations when using the AxoSyslog elasticsearch2 destination:
Since AxoSyslog uses Java libraries, the elasticsearch2 destination has significant memory usage.
Example: Sending log data to Elasticsearch version 2.x and above
The following example defines an elasticsearch2 destination that sends messages in transport mode to an Elasticsearch server running on the localhost, using only the required parameters.
Verify the certificate of the Elasticsearch server and perform certificate authentication (this is actually a mutual, certificate-based authentication between the AxoSyslog client and the Elasticsearch server):
The elasticsearch2() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of AxoSyslog. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the elasticsearch configuration snippet on GitHub. For details on extending AxoSyslog in Java, see the Getting started with syslog-ng development guide.
Note
If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again.
8.5.1 - Prerequisites
To send messages from AxoSyslog to Elasticsearch, complete the following steps.
Steps:
Download and install the Java Runtime Environment (JRE), 2.x (or newer). The AxoSyslogelasticsearch2 destination is tested and supported when using the Oracle implementation of Java. Other implementations are untested and unsupported, they may or may not work as expected.
Note
This step is only required if you use the elasticsearch2 destination in node mode or transport mode.
This step is only required if you use the elasticsearch2 destination in node mode or transport mode.
Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/elasticsearch/lib/) where AxoSyslog can access them. You must specify this directory in the AxoSyslog configuration file. The files are located in the lib directory and its subdirectories of the Elasticsearch release package.
8.5.2 - How AxoSyslog interacts with Elasticsearch
The AxoSyslog application sends the log messages to the official Elasticsearch client library, which forwards the data to the Elasticsearch nodes. The way AxoSyslog interacts with Elasticsearch is described in the following steps.
After AxoSyslog is started and the first message arrives to the elasticsearch2 destination, the elasticsearch2 destination tries to connect to the Elasticsearch server or cluster. If the connection fails, AxoSyslog will repeatedly attempt to connect again after the period set in time-reopen() expires.
If the connection is established, AxoSyslog sends JSON-formatted messages to Elasticsearch.
If flush-limit is set to 1: AxoSyslog sends the message reliably: it sends a message to Elasticsearch, then waits for a reply from Elasticsearch. In case of failure, AxoSyslog repeats sending the message, as set in the retries() parameter. If sending the message fails for retries() times, AxoSyslog drops the message.
This method ensures reliable message transfer, but is slow (about 1000 messages/second).
If flush-limit is higher than 1: AxoSyslog sends messages in a batch, and receives the response asynchronously. In case of a problem, AxoSyslog cannot resend the messages.
This method is relatively fast (depending on the size of flush-limit, about 8000 messages/second), but the transfer is not reliable. In transport mode, over 5000-30000 messages can be lost before AxoSyslog recognizes the error. In node mode, about 1000 messages can be lost.
If concurrent-requests is higher than 1, AxoSyslog can send multiple batches simultaneously, increasing performance (and also the number of messages that can be lost in case of an error).
Version 3.10 and newer of AxoSyslog automatically converts the timestamp (date) of the message to UTC, as needed by Elasticsearch and Kibana.
8.5.3 - Client modes
The AxoSyslog application can interact with Elasticsearch in the following modes of operation: http, https, node, searchguard, and transport.
HTTP mode
The AxoSyslog application sends messages over HTTP using the REST API of Elasticsearch, and uses the cluster-url() and cluster() options from the AxoSyslog configuration file. In HTTP mode, AxoSyslogelasticsearch2 driver can send log messages to every Elasticsearch version, including 1.x-6.x. Note that HTTP mode is available in AxoSyslog version 3.8 and newer.
In version 3.10 and newer, you can list multiple servers in HTTP and HTTPS mode in the cluster-url() and server() options. The AxoSyslog application will use these destination servers in load-balancing fashion. Note that load-balancing is handled by an external library (Jest), AxoSyslog does not have any direct influence on it.
HTTPS mode
The AxoSyslog application sends messages over an encrypted and optionally authenticated HTTPS channel using the REST API of Elasticsearch, and uses the cluster-url() and cluster() options from the AxoSyslog configuration file. In HTTPS mode, AxoSyslogelasticsearch2 driver can send log messages to every Elasticsearch version, including 1.x-6.x. Note that HTTPS mode is available in AxoSyslog version 3.10 and newer.
This mode supports password-based and certificate-based authentication of the client, and can verify the certificate of the server as well.
In version 3.10 and newer, you can list multiple servers in HTTP and HTTPS mode in the cluster-url() and server() options. The AxoSyslog application will use these destination servers in load-balancing fashion. Note that load-balancing is handled by an external library (Jest), AxoSyslog does not have any direct influence on it.
Transport mode
The AxoSyslog application uses the transport client API of Elasticsearch, and uses the server(), port(), and cluster() options from the AxoSyslog configuration file.
Node mode
The AxoSyslog application acts as an Elasticsearch node (client no-data), using the node client API of Elasticsearch. Further options for the node can be describe in an Elasticsearch configuration file specified in the resource() option.
Note
In Node mode, it is required to define the home of the elasticsearch installation with the <code>path.home</code> parameter in the <code>.yml</code> file. For example: <code>path.home: /usr/share/elasticsearch</code>.
Search Guard mode
Use the Search Guard Elasticsearch plugin to encrypt and authenticate your connections from AxoSyslog to Elasticsearch 2.x. For Elasticsearch versions 5.x and newer, use HTTPS mode. For details on configuring Search Guard mode, see Search Guard.
8.5.4 - Search Guard
Purpose:
Version 3.9 and later supports the Search Guard Elasticsearch plugin (version 2.4.1.16 and newer) to encrypt and authenticate your connections to from AxoSyslog to Elasticsearch 2 and newer. To configure AxoSyslog to send messages to an Elasticsearch 2.x cluster that uses Search Guard, complete the following steps.
To connect to an Elasticsearch 5.x or newer cluster, use HTTPS mode.
Steps:
Install the Search Guard plugin on your AxoSyslog host. Use the plugin version that matches the version of your Elasticsearch installation.
Create a certificate for your AxoSyslog host, and add the certificate to the SYSLOG_NG-NODE_NAME-keystore.jks file. You can configure the location of this file in the Elasticsearch resources file under the path.conf parameter. For details, see the Search Guard documentation.
Configure an Elasticsearch destination in AxoSyslog that uses the searchguard client mode. For example:
Configure the Elasticsearch resource file (for example, /etc/syslog-ng/elasticsearch.yml) as needed for your environment. Note the searchguard: section.
The elasticsearch2 destination can directly send log messages to Elasticsearch, allowing you to search and analyze your data in real time, and visualize it with Kibana. The elasticsearch2 destination has the following options.
Required options:
The following options are required: index(), type(). In node mode, either the cluster() or the resource() option is required as well. Note that to use elasticsearch2, you must add the following lines to the beginning of your AxoSyslog configuration:
@include "scl.conf"
client-lib-dir()
Type:
string
Default:
The AxoSyslog module directory: /opt/syslog-ng/lib/syslog-ng/java-modules/
Description: The list of the paths where the required Java classes are located. For example, class-path("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/my-java-libraries/libs/"). If you set this option multiple times in your AxoSyslog configuration (for example, because you have multiple Java-based destinations), AxoSyslog will merge every available paths to a single list.
Description: Include the path to the directory where you copied the required libraries (see Prerequisites), for example, client-lib-dir(/user/share/elasticsearch-2.2.0/lib).
client-mode()
Type:
http
Default:
node
Description: Specifies the client mode used to connect to the Elasticsearch server, for example, client-mode("node").
HTTP mode
The AxoSyslog application sends messages over HTTP using the REST API of Elasticsearch, and uses the cluster-url() and cluster() options from the AxoSyslog configuration file. In HTTP mode, AxoSyslogelasticsearch2 driver can send log messages to every Elasticsearch version, including 1.x-6.x. Note that HTTP mode is available in AxoSyslog version 3.8 and newer.
In version 3.10 and newer, you can list multiple servers in HTTP and HTTPS mode in the cluster-url() and server() options. The AxoSyslog application will use these destination servers in load-balancing fashion. Note that load-balancing is handled by an external library (Jest), AxoSyslog does not have any direct influence on it.
HTTPS mode
The AxoSyslog application sends messages over an encrypted and optionally authenticated HTTPS channel using the REST API of Elasticsearch, and uses the cluster-url() and cluster() options from the AxoSyslog configuration file. In HTTPS mode, AxoSyslogelasticsearch2 driver can send log messages to every Elasticsearch version, including 1.x-6.x. Note that HTTPS mode is available in AxoSyslog version 3.10 and newer.
This mode supports password-based and certificate-based authentication of the client, and can verify the certificate of the server as well.
In version 3.10 and newer, you can list multiple servers in HTTP and HTTPS mode in the cluster-url() and server() options. The AxoSyslog application will use these destination servers in load-balancing fashion. Note that load-balancing is handled by an external library (Jest), AxoSyslog does not have any direct influence on it.
Transport mode
The AxoSyslog application uses the transport client API of Elasticsearch, and uses the server(), port(), and cluster() options from the AxoSyslog configuration file.
Node mode
The AxoSyslog application acts as an Elasticsearch node (client no-data), using the node client API of Elasticsearch. Further options for the node can be describe in an Elasticsearch configuration file specified in the resource() option.
Note
In Node mode, it is required to define the home of the elasticsearch installation with the <code>path.home</code> parameter in the <code>.yml</code> file. For example: <code>path.home: /usr/share/elasticsearch</code>.
Search Guard mode
Use the Search Guard Elasticsearch plugin to encrypt and authenticate your connections from AxoSyslog to Elasticsearch 2.x. For Elasticsearch versions 5.x and newer, use HTTPS mode. For details on configuring Search Guard mode, see Search Guard.
cluster()
Type:
string
Default:
N/A
Description: Specifies the name or the Elasticsearch cluster, for example, cluster("my-elasticsearch-cluster"). Optionally, you can specify the name of the cluster in the Elasticsearch resource file. For details, see resource().
cluster-url()
Type:
string
Default:
N/A
Description: Specifies the URL or the Elasticsearch cluster, for example, cluster-url("http://192.168.10.10:9200")").
In version 3.10 and newer, you can list multiple servers in HTTP and HTTPS mode in the cluster-url() and server() options. The AxoSyslog application will use these destination servers in load-balancing fashion. Note that load-balancing is handled by an external library (Jest), AxoSyslog does not have any direct influence on it.
Description: The number of concurrent (simultaneous) requests that AxoSyslog sends to the Elasticsearch server. Set this option to 1 or higher to increase performance. When using the concurrent-requests() option, make sure that the flush-limit() option is higher than one, otherwise it will not have any noticeable effect. For details, see flush-limit().
Warning
Hazard of data loss! Using the concurrent-requests() option increases the number of messages lost in case the Elasticsearch server becomes unaccessible.
custom-id()
Type:
template or template function
Default:
N/A
Description: Use this option to specify a custom ID for the records inserted into Elasticsearch. If this option is not set, the Elasticsearch server automatically generates and ID for the message. For example: custom-id(${UNIQID}) (Note that to use the ${UNIQID} macro, the use-uniqid() global option must be enabled. For details, see use-uniqid().)
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: The number of messages that AxoSyslog sends to the Elasticsearch server in a single batch.
If flush-limit is set to 1: AxoSyslog sends the message reliably: it sends a message to Elasticsearch, then waits for a reply from Elasticsearch. In case of failure, AxoSyslog repeats sending the message, as set in the retries() parameter. If sending the message fails for retries() times, AxoSyslog drops the message.
This method ensures reliable message transfer, but is slow (about 1000 messages/second).
If flush-limit is higher than 1: AxoSyslog sends messages in a batch, and receives the response asynchronously. In case of a problem, AxoSyslog cannot resend the messages.
This method is relatively fast (depending on the size of flush-limit, about 8000 messages/second), but the transfer is not reliable. In transport mode, over 5000-30000 messages can be lost before AxoSyslog recognizes the error. In node mode, about 1000 messages can be lost.
If concurrent-requests is higher than 1, AxoSyslog can send multiple batches simultaneously, increasing performance (and also the number of messages that can be lost in case of an error).
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Determines how AxoSyslog authenticates to the Elasticsearch server. Depending on the value of this option, you might have to set other options as well. Possible values:
none: Connect to the Elasticsearch server without authentication.
Verify the certificate of the Elasticsearch server and perform certificate authentication (this is actually a mutual, certificate-based authentication between the AxoSyslog client and the Elasticsearch server):
Description: The password to use for password-authentication on the Elasticsearch server. You must also set the http-auth-type-basic-username option.
This option is used only in HTTPS mode with basic authentication: client-mode("https") and http-auth-type("basic"), and is available in AxoSyslog version 3.10 and newer.
Description: The username to use for password-authentication on the Elasticsearch server. You must also set the http-auth-type-basic-password option.
This option is used only in HTTPS mode with basic authentication: client-mode("https") and http-auth-type("basic"), and is available in AxoSyslog version 3.10 and newer.
Description: Name of the Elasticsearch index to store the log messages. You can use macros and templates as well.
java-keystore-filepath()
Type:
string
Default:
N/A
Description: Path to the Java keystore file that stores the certificate that AxoSyslog uses to authenticate on the Elasticsearch server. You must also set the java-keystore-password option.
To import a certificate into a Java keystore, use the appropriate tool of your Java implementation. For example, on Oracle Java, you can use the keytool utility:
keytool -import -alias ca -file <certificate-to-import> -keystore <keystore-to-import> -storepass <password-to-the-keystore>
This option is used only in HTTPS mode with basic authentication: client-mode("https") and http-auth-type("clientcert"), and is available in AxoSyslog version 3.10 and newer.
Verify the certificate of the Elasticsearch server and perform certificate authentication (this is actually a mutual, certificate-based authentication between the AxoSyslog client and the Elasticsearch server):
Description: The password of the Java keystore file set in the java-keystore-filepath option.
To import a certificate into a Java keystore, use the appropriate tool of your Java implementation. For example, on Oracle Java, you can use the keytool utility:
keytool -import -alias ca -file <certificate-to-import> -keystore <keystore-to-import> -storepass <password-to-the-keystore>
This option is used only in HTTPS mode with basic authentication: client-mode("https") and http-auth-type("clientcert"), and is available in AxoSyslog version 3.10 and newer.
Verify the certificate of the Elasticsearch server and perform certificate authentication (this is actually a mutual, certificate-based authentication between the AxoSyslog client and the Elasticsearch server):
Description: Path to the Java keystore file that stores the CA certificate that AxoSyslog uses to verify the certificate of the Elasticsearch server. You must also set the java-truststore-password option.
If you do not set the java-truststore-filepath option, AxoSyslog does accepts any certificate that the Elasticsearch server shows. In this case, the identity of the server is not verified, only the connection is encrypted.
To import a certificate into a Java keystore, use the appropriate tool of your Java implementation. For example, on Oracle Java, you can use the keytool utility:
keytool -import -alias ca -file <certificate-to-import> -keystore <keystore-to-import> -storepass <password-to-the-keystore>
This option is used only in HTTPS mode: client-mode("https"), and is available in AxoSyslog version 3.10 and newer.
Verify the certificate of the Elasticsearch server without authentication:
Verify the certificate of the Elasticsearch server and perform certificate authentication (this is actually a mutual, certificate-based authentication between the AxoSyslog client and the Elasticsearch server):
Description: The password of the Java truststore file set in the java-truststore-filepath option.
To import a certificate into a Java keystore, use the appropriate tool of your Java implementation. For example, on Oracle Java, you can use the keytool utility:
keytool -import -alias ca -file <certificate-to-import> -keystore <keystore-to-import> -storepass <password-to-the-keystore>
This option is used only in HTTPS mode: client-mode("https"), and is available in AxoSyslog version 3.10 and newer.
Verify the certificate of the Elasticsearch server without authentication:
Verify the certificate of the Elasticsearch server and perform certificate authentication (this is actually a mutual, certificate-based authentication between the AxoSyslog client and the Elasticsearch server):
Description: Specify the Java Virtual Machine (JVM) settings of your Java destination from the AxoSyslog configuration file.
For example:
jvm-options("-Xss1M -XX:+TraceClassLoading")
You can set this option only as a global option, by adding it to the options statement of the syslog-ng.conf configuration file.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
on-error()
Type:
One of: drop-message, drop-property, fallback-to-string, silently-drop-message, silently-drop-property, silently-fallback-to-string
Default:
Use the global setting (which defaults to drop-message)
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
port()
Type:
number
Default:
9300
Description: The port number of the Elasticsearch server. This option is used only in transport mode: client-mode("transport")
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
resource()
Type:
string
Default:
N/A
Description: The list of Elasticsearch resources to load, separated by semicolons. For example, resource("/home/user/elasticsearch/elasticsearch.yml;/home/user/elasticsearch/elasticsearch2.yml").
server()
Type:
list of hostnames
Default:
127.0.0.1
Description: Specifies the hostname or IP address of the Elasticsearch server. When specifying an IP address, IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well. When specifying multiple addresses, use space to separate the addresses, for example, server("127.0.0.1 remote-server-hostname1 remote-server-hostname2")
This option is used only in transport mode: client-mode("transport")
In version 3.10 and newer, you can list multiple servers in HTTP and HTTPS mode in the cluster-url() and server() options. The AxoSyslog application will use these destination servers in load-balancing fashion. Note that load-balancing is handled by an external library (Jest), AxoSyslog does not have any direct influence on it.
Description: By default, when connecting to an Elasticsearch cluster, AxoSyslog checks the state of the cluster. If the primary shards of the cluster are not active, AxoSyslog will not send messages, but wait for them to become active. To disable this health check and send the messages to Elasticsearch anyway, use the skip-cluster-health-check(yes) option in your configuration.
template()
Type:
template or template function
Default:
$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})
Description: The message as sent to the Elasticsearch server. Typically, you will want to use the command-line notation of the format-json template function.
To add a @timestamp field to the message, for example, to use with Kibana, include the @timestamp=${ISODATE} expression in the template. For example: template($(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}))
For details on formatting messages in JSON format, see format-json.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
Version 3.10 and newer of AxoSyslog automatically converts the timestamp (date) of the message to UTC, as needed by Elasticsearch and Kibana.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
type()
Type:
string
Default:
N/A
Description: The type of the index. For example, type("test").
8.6 - elasticsearch-http: Send messages to Elasticsearch HTTP Bulk API
Version 3.21 of AxoSyslog can directly post log messages to an Elasticsearch deployment using the Elasticsearch Bulk API over the HTTP and Secure HTTP (HTTPS) protocols.
HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format.
Use an empty string to omit the type from the index: type(""). For example, you need to do that when using Elasticsearch 7 or newer, and you use a mapping in Elasticsearch to modify the type of the data.
You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.
Alternatively, you can leave the HTTP as-is, in which case the driver leaves the default http_proxy and https_proxy environment variables unmodified.
This driver is actually a reusable configuration snippet configured to send log messages using the tcp() driver using a template. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
8.6.1 - Batch mode and load balancing
The elasticsearch-http() destination automatically sends multiple log messages in a single HTTP request, increasing the rate of messages that your Elasticsearch deployment can consume. For details on adjusting and fine-tuning the batch mode of the elasticsearch-http() destination, see the following section.
Batch size
The batch-bytes(), batch-lines(), and batch-timeout() options of the destination determine how many log messages AxoSyslog sends in a batch. The batch-lines() option determines the maximum number of messages AxoSyslog puts in a batch in. This can be limited based on size and time:
AxoSyslog sends a batch every batch-timeout() milliseconds, even if the number of messages in the batch is less than batch-lines(). That way the destination receives every message in a timely manner even if suddenly there are no more messages.
AxoSyslog sends the batch if the total size of the messages in the batch reaches batch-bytes() bytes.
To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-bytes(), batch-lines(), batch-timeout() options.
Example: HTTP batch mode
In the following example, a batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).
Load balancing between multiple Elasticsearch indexers
Starting with version 3.19, you can specify multiple URLs, for example, url("site1" "site2"). In this case, AxoSyslog sends log messages to the specified URLs in a load-balance fashion. This means that AxoSyslog sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order AxoSyslog has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.
Warning
If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.
Starting with version AxoSyslog version 3.22, you can use any of the following formats to specify multiple URLs:
url("server1", "server2", "server3");# comma-separated strings url("server1""server2""server3");# space-separated strings url("server1 server2 server3");# space-separated within a single string
Example: HTTP load balancing
The following destination sends log messages to 3 different Elasticsearch indexer nodes. Each node is assigned a separate worker thread. A batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).
If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1", "site2", "site3")), set the workers() option to 3 or more.
8.6.2 - elasticsearch-http() destination options
The elasticsearch-http destination of AxoSyslog can directly post log messages to an Elasticsearch deployment using the Elasticsearch Bulk API over the HTTP and Secure HTTP (HTTPS) protocols. The elasticsearch-http destination has the following options. The required options are: index(), type(), and url().
This destination is available in AxoSyslog version 3.21 and later.
batch-bytes()
Accepted values:
number [bytes]
Default:
none
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Name of a file that contains an X.509 CA certificate (or a certificate chain) in PEM format. The AxoSyslog application uses this certificate to validate the certificate of the HTTPS server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Name of a file, that contains an X.509 certificate (or a certificate chain) in PEM format, suitable as a TLS certificate, matching the private key set in the key-file() option. The AxoSyslog application uses this certificate to authenticate the AxoSyslog client on the destination server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Depends on the OpenSSL version that AxoSyslog uses
Description: Specifies the cipher, hash, and key-exchange algorithms used for the encryption, for example, ECDHE-ECDSA-AES256-SHA384. The list of available algorithms depends on the version of OpenSSL used to compile AxoSyslog. To specify multiple ciphers, separate the cipher names with a colon, and enclose the list between double-quotes, for example:
For a list of available algorithms, execute the openssl ciphers -v command. The first column of the output contains the name of the algorithms to use in the cipher-suite() option, the second column specifies which encryption protocol uses the algorithm (for example, TLSv1.2). That way, the cipher-suite() also determines the encryption protocol used in the connection: to disable SSLv3, use an algorithm that is available only in TLSv1.2, and that both the client and the server supports. You can also specify the encryption protocols using ssl-options().
You can also use the following command to automatically list only ciphers permitted in a specific encryption protocol, for example, TLSv1.2:
echo"cipher-suite(\"$(openssl ciphers -v | grep TLSv1.2 | awk '{print $1}'| xargs echo -n | sed 's/ /:/g'| sed -e 's/:$//')\")"
Note that starting with version 3.10, when AxoSyslog receives TLS-encrypted connections, the order of ciphers set on the AxoSyslog server takes precedence over the client settings.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Sets the specified value as the ID of the Elasticsearch index (_id).
delimiter()
Accepted values:
string
Default:
newline character
Description: By default, AxoSyslog separates the log messages of the batch with a newline character. You can specify a different delimiter by using the delimiter() option.
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The name of the Elasticsearch index where Elasticsearch will store the messages received from AxoSyslog. This option is mandatory for this destination.
You can use macros and template functions, but you must ensure that the resolved template contains only characters that Elasticsearch permits in the name of the index. The AxoSyslog application does not validate the name of the index. For details on the characters permitted in the name of Elasticsearch indices, see the documentation of Elasticsearch.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
key-file()
Accepted values:
Filename
Default:
none
Description: The name of a file that contains an unencrypted private key in PEM format, suitable as a TLS key. If properly configured, the AxoSyslog application uses this private key and the matching certificate (set in the cert-file() option) to authenticate the AxoSyslog client on the destination server.
This destination supports only unencrypted key files (that is, the private key cannot be password-protected).
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: The password that AxoSyslog uses to authenticate on the server where it sends the messages.
peer-verify()
Accepted values:
`yes
Default:
yes
Description: Verification method of the peer. The following table summarizes the possible options and their results depending on the certificate of the peer.
The remote peer has:
no certificate
invalid certificate
valid certificate
Local peer-verify() setting
no (optional-untrusted)
TLS-encryption
TLS-encryption
TLS-encryption
yes (required-trusted)
rejected connection
rejected connection
TLS-encryption
For untrusted certificates only the existence of the certificate is checked, but it does not have to be valid — AxoSyslog accepts the certificate even if it is expired, signed by an unknown CA, or its CN and the name of the machine mismatches.
Warning
When validating a certificate, the entire certificate chain must be valid, including the CA certificate. If any certificate of the chain is invalid, AxoSyslog will reject the connection.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
To handle HTTP error responses, if the HTTP server returns 5xx codes, AxoSyslog will attempt to resend messages until the number of attempts reaches retries. If the HTTP server returns 4xx codes, AxoSyslog will drop the messages.
ssl-version()
Type:
string
Default:
None, uses the libcurl default
Description: Specifies the permitted SSL/TLS version. Possible values: sslv2, sslv3, tlsv1, tlsv1_0, tlsv1_1, tlsv1_2, tlsv1_3.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
type()
Type:
string or template
Default:
N/A
Description: The type of the Elasticsearch index.
Use an empty string to omit the type from the index: type(""). For example, you need to do that when using Elasticsearch 7 or newer, and you use a mapping in Elasticsearch to modify the type of the data.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
timeout()
Type:
number [seconds]
Default:
10
Description: The value (in seconds) to wait for an operation to complete, and attempt to reconnect the server if exceeded.
url()
Type:
URL or list of URLs, for example, url(“site1” “site2”)
Default:
N/A
Description: Specifies the hostname or IP address and optionally the port number of the Elasticsearch indexer. Use a colon (:) after the address to specify the port number of the server. For example: http://your-elasticsearch-indexer.server:8088/_bulk
This option is mandatory for this destination.
Make sure that the URL ends with _bulk, this is the Elasticsearch API endpoint that properly parses the messages sent by AxoSyslog.
In case the server on the specified URL returns a redirect request, AxoSyslog automatically follows maximum 3 redirects. Only HTTP and HTTPS based redirections are supported.
Starting with version 3.19, you can specify multiple URLs, for example, url("site1" "site2"). In this case, AxoSyslog sends log messages to the specified URLs in a load-balance fashion. This means that AxoSyslog sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order AxoSyslog has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.
Warning
If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.
Starting with version AxoSyslog version 3.22, you can use any of the following formats to specify multiple URLs:
url("server1", "server2", "server3");# comma-separated strings url("server1""server2""server3");# space-separated strings url("server1 server2 server3");# space-separated within a single string
user()
Type:
string
Default:
Description: The username that AxoSyslog uses to authenticate on the server where it sends the messages.
use-system-cert-store()
Type:
`yes
Default:
no
Description: Use the certificate store of the system for verifying HTTPS certificates. For details, see the curl documentation.
workers()
Type:
integer
Default:
4
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1", "site2", "site3")), set the workers() option to 3 or more.
8.7 - file: Store messages in plain-text files
The file driver is one of the most important destination drivers. It allows to output messages to the specified text file, or to a set of files.
The destination filename may include macros which get expanded when the message is written, thus a simple file() driver may create several files: for example, AxoSyslog can store the messages of client hosts in a separate file for each host. For more information on available macros see Macros of AxoSyslog.
If the expanded filename refers to a directory which does not exist, it will be created depending on the create-dirs() setting (both global and a per destination option).
The file() has a single required parameter that specifies the filename that stores the log messages. For the list of available optional parameters, see file() destination options.
Declaration:
file(filename options());
Example: Using the file() driver
destination d_file { file("/var/log/messages");};
Example: Using the file() driver with macros in the file name and a template for the message
When using this destination, update the configuration of your log rotation program to rotate these files. Otherwise, the log files can become very large.
Also, after rotating the log files, reload AxoSyslog using the syslog-ng-ctl reload command, or use another method to send a SIGHUP to AxoSyslog.
Warning
Since the state of each created file must be tracked by syslog-ng, it consumes some memory for each file. If no new messages are written to a file within 60 seconds (controlled by the time-reap() global option), it is closed, and its state is freed.
Exploiting this, a DoS attack can be mounted against the system. If the number of possible destination files and its needed memory is more than the amount available on the AxoSyslog server.
The most suspicious macro is ${PROGRAM}, where the number of possible variations is rather high. Do not use the ${PROGRAM} macro in insecure environments.
8.7.1 - file() destination options
The file() driver outputs messages to the specified text file, or to a set of files. The file() destination has the following options:
Warning
When creating several thousands separate log files, AxoSyslog might not be able to open the required number of files. This might happen for example, when using the ${HOST} macro in the filename while receiving messages from a large number of hosts. To overcome this problem, adjust the --fd-limit command-line parameter of AxoSyslog or the global ulimit parameter of your host. For setting the --fd-limit command-line parameter ofAxoSyslog see the The syslog-ng manual page. For setting the ulimit parameter of the host, see the documentation of your operating system.
create-dirs()
Type:
yes or no
Default:
no
Description: Enable creating non-existing directories when creating files or socket files.
dir-group()
Type:
string
Default:
Use the global settings
Description: The group of the directories created by syslog-ng. To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-group().
dir-owner()
Type:
string
Default:
Use the global settings
Description: The owner of the directories created by syslog-ng. To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-owner().
Starting with version 3.16, the default value of this option is -1, so AxoSyslog does not change the ownership, unless explicitly configured to do so.
dir-perm()
Type:
number
Default:
Use the global settings
Description: The permission mask of directories created by syslog-ng. Log directories are only created if a file after macro expansion refers to a non-existing directory, and directory creation is enabled (see also the create-dirs() option). For octal numbers prefix the number with 0, for example, use 0755 for rwxr-xr-x.
To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-perm(). Note that when creating a new directory without specifying attributes for dir-perm(), the default permission of the directories is masked with the umask of the parent process (typically 0022).
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
threaded: The threaded flag enables multithreading for the destination. For details on multithreading, see Multithreading and scaling.
Note
The file destination uses multiple threads only if the destination filename contains macros.
flush-lines()
Type:
number
Default:
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
fsync()
Type:
yes or no
Default:
no
Description: Forces an fsync() call on the destination fd after each write.
Note
Enabling this option may seriously degrade performance.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Set the group of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: group().
local-time-zone()
Type:
name of the timezone, or the timezone offset
Default:
The local timezone.
Description: Sets the timezone used when expanding filename and tablename templates.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
overwrite-if-older()
Type:
number (seconds)
Default:
0
Description: If set to a value higher than 0, AxoSyslog checks when the file was last modified before starting to write into the file. If the file is older than the specified amount of time (in seconds), then AxoSyslog removes the existing file and opens a new file with the same name. In combination with for example, the ${WEEKDAY} macro, this can be used for simple log rotation, in case not all history has to be kept. (Note that in this weekly log rotation example if its Monday 00:01, then the file from last Monday is not seven days old, because it was probably last modified shortly before 23:59 last Monday, so it is actually not even six days old. So in this case, set the overwrite-if-older() parameter to a-bit-less-than-six-days, for example, to 518000 seconds.
owner()
Type:
string
Default:
Use the global settings
Description: Set the owner of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: owner().
pad-size()
Type:
number
Default:
0
Description: If set, AxoSyslog will pad output messages to the specified size (in bytes). Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes).
Warning
Hazard of data loss! If the size of the incoming message is larger than the previously set pad-size() value, AxoSyslog will truncate the message to the specified size. Therefore, all message content above that size will be lost.
perm()
Type:
number
Default:
Use the global settings
Description: The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the number with 0, for example, use 0755 for rwxr-xr-x.
To preserve the original properties of an existing file, use the option without specifying an attribute: perm().
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
symlink-as()
Type:
Filename
Default:
N/A
Description: The configured file name will be used as a symbolic link to the last created file by file destination.
In this case the /var/log/cron should point to the current month.
Note
The symlink uses the same permissions as the file destination.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
time-reap()
Accepted values:
number (seconds)
Default:
60 or 0, see description for details
Description: The time to wait in seconds before an idle destination file or pipe is closed. Note that only destination files having macros in their filenames are closed automatically.
Starting with version 3.23, the way how time-reap() works is the following.
If the time-reap() option of the destination is set, that value is used, for example:
destination d_fifo { pipe("/tmp/test.fifo",
time-reap(30)# sets time-reap() for this destination only);};
If the time-reap() option of the destination is not set, and the destination does not use a template or macro in its filename or path, time-reap() is automatically set to 0. For example:
destination d_fifo { pipe("/tmp/test.fifo",
);};
Otherwise, the value of the global time-reap() option is used, which defaults to 60 seconds.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
This driver is actually a reusable configuration snippet configured to send log messages using the http() driver using a template. You can find the source of this configuration snippet on GitHub.
Options
The following options are specific to the google-pubsub() destination. But since this destination is based on the http() destination, you can use the options of the http() destination as well if needed.
Note: The google-pubsub() destination automatically configures some of these http() destination options as required by the Google Pub/Sub API.
attributes()
Type:
string
Default:
"--scope rfc5424,all-nv-pairs --exclude MESSAGE"
Description: A JSON object representing key-value pairs for the Pub/Sub Event, formatted as AxoSyslog value-pairs. By default, the google-pubsub() destination sends the RFC5424 fields as attributes. If you want to send different fields, override the default template. By default, the message part is sent in the data() option.
Description: The template to use as the data part of the Google Pub/Sub message.
project()
Type:
string
Default:
-
Description: The ID of the Google Cloud project where AxoSyslog sends the data. The Pub/Sub API must be enabled for the project.
topic()
Type:
string
Default:
-
Description: The name of the Google Pub/Sub topic where AxoSyslog sends the data.
8.9 - graphite: Send metrics to Graphite
The graphite() destination can send metrics to a Graphite server to store numeric time-series data. There are many ways to feed the Graphite template function with name value pairs. The AxoSyslog CSV and PatternDB parsers (for details, see Using pattern parsers) can parse log messages and generate name value pairs based on message content. The CSV parser (for details, see Parsing messages with comma-separated and similar values) can be used for logs that have a constant field based structure, like the Apache web server access logs. The [patterndb() parser] can parse information and can extract important fields from free form log messages, as long as patterns describing the log messages are available. Another way is to send JSON-based log messages (for details, see JSON parser) to AxoSyslog, like running a simple shell script collecting metrics and running it from cron regularly.
To use the graphite() destination, the only mandatory parameter is payload, which specifies the value pairs to send to graphite. In the following example any value pairs starting with “monitor." are forwarded to graphite.
The graphite() destination is only a wrapper around the network() destination and the graphite-output template function. If you want to fine-tune the TCP parameters, use the network() destination instead, as described in graphite-output.
8.9.1 - graphite() destination options
The graphite() destination has the following options:
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The hostname or IP address of the Graphite server.
port()
Type:
number
Default:
2003
Description: The port number of the Graphite server.
payload()
Type:
parameter list of the payload() option
Default:
empty string
Description: The payload() option allows you to select which value pairs to forward to graphite.
The syntax of payload is different from the syntax of value-pairs(): use the command-line syntax used in the format-json template function. For details on using the payload() option, see graphite-output.
Note
If left empty, there is no data to be forwarded to Graphite.
8.10 - graylog2: Send logs to Graylog
graylog2(): Sending logs to Graylog
You can use the graylog2() destination and a Graylog Extended Log Format (GELF) template to send syslog messages to Graylog.
You can forward simple name-value pairs where the name starts with a dot or underscore. If names of your name-value pairs include dots other than the first character, you should use JSON formatting directly instead of the GELF template and send logs to a raw TCP port in Graylog, which can then extract fields from nested JSON. Version 3.21 and later also supports TLS-encrypted connection to the Graylog server.
Declaration:
graylog2();
Example: Using the graylog2() driver
You can send syslog messages to Graylog using the graylog2() destination. The graylog2() destination uses the GELF template, the native data format of Graylog.
On the Graylog side, configure a GELF TCP input. For more information, see the relevant Graylog documentation.
On the AxoSyslog side, configure the name or IP address of the host running Graylog.
If you parsed your messages using syslog-ng, the template also forwards any name-value pairs where the name starts with a dot or underscore.
Note
If there is a dot in a field name other than the first character, AxoSyslog creates nested JSON while formatting the message. Nested JSON is not automatically parsed in GELF messages.
Sending nested JSON to Graylog
While sending nested JSON inside GELF is possible, it is not convenient. If you use parsing and normalization and dot notation in field names, use pure JSON instead of GELF to forward your messages.
On the Graylog side, create a new raw TCP input.
Still in Graylog, once the raw TCP input is ready, add a JSON extractor to it.
On the AxoSyslog side, use a network destination combined with a template utilizing format-json as shown in the example below:
The graylog2() destination has the following options:
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
transport()
Type:
udp, tcp, or tls
Default:
tcp
Description: Specifies the protocol used to send messages to the destination server.
If you use the udp transport, AxoSyslog automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.
8.11 - hdfs: Store messages on the Hadoop Distributed File System (HDFS)
Starting with version 3.7, AxoSyslog can send plain-text log files to the Hadoop Distributed File System (HDFS), allowing you to store your log data on a distributed, scalable file system. This is especially useful if you have huge amounts of log messages that would be difficult to store otherwise, or if you want to process your messages using Hadoop tools (for example, Apache Pig).
Note the following limitations when using the AxoSyslog hdfs destination:
Since AxoSyslog uses the official Java HDFS client, the hdfs destination has significant memory usage (about 400MB).
You cannot set when log messages are flushed. Hadoop performs this action automatically, depending on its configured block size, and the amount of data received. There is no way for the AxoSyslog application to influence when the messages are actually written to disk. This means that AxoSyslog cannot guarantee that a message sent to HDFS is actually written to disk. When using flow-control, AxoSyslog acknowledges a message as written to disk when it passes the message to the HDFS client. This method is as reliable as your HDFS environment.
The hdfs() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of AxoSyslog. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the hdfs configuration snippet on GitHub. For details on extending AxoSyslog in Java, see the Getting started with syslog-ng development guide.
Note
If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again.
8.11.1 - Prerequisites
To send messages from AxoSyslog to HDFS, complete the following steps.
Steps:
If you want to use the Java-based modules of AxoSyslog (for example, the Elasticsearch, HDFS, or Kafka destinations), you must compile AxoSyslog with Java support.
Download and install the Java Runtime Environment (JRE), 1.7 (or newer). You can use OpenJDK or Oracle JDK, other implementations are not tested.
Set LD_LIBRARY_PATH to include the libjvm.so file, for example:LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH
Note that many platforms have a simplified links for Java libraries. Use the simplified path if available. If you use a startup script to start AxoSyslog set LD_LIBRARY_PATH in the script as well.
If you are behind an HTTP proxy, create a gradle.properties under the modules/java-modules/ directory. Set the proxy parameters in the file. For details, see The Gradle User Guide.
Extract the HDFS libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/hadoop/lib/) where AxoSyslog can access them. You must specify this directory in the AxoSyslog configuration file. The files are located in the various lib directories under the share/ directory of the Hadoop release package. (For example, in Hadoop 2.7, required files are common/hadoop-common-2.7.0.jar, common/libs/\*.jar, hdfs/hadoop-hdfs-2.7.0.jar, hdfs/lib/\*, but this may change between Hadoop releases, so it is easier to copy every .jar file into a single directory.
8.11.2 - How AxoSyslog interacts with HDFS
The AxoSyslog application sends the log messages to the official HDFS client library, which forwards the data to the HDFS nodes. The way AxoSyslog interacts with HDFS is described in the following steps.
After AxoSyslog is started and the first message arrives to the hdfs destination, the hdfs destination tries to connect to the HDFS NameNode. If the connection fails, AxoSyslog will repeatedly attempt to connect again after the period set in time-reopen() expires.
AxoSyslog checks if the path to the logfile exists. If a directory does not exist AxoSyslog automatically creates it. AxoSyslog creates the destination file (using the filename set in the AxoSyslog configuration file, with a UUID suffix to make it unique, for example, /usr/hadoop/logfile.txt.3dc1c59e-ab3b-4b71-9e81-93db477ed9d9) and writes the message into the file. After the file is created, AxoSyslog will write all incoming messages into the hdfs destination.
Note
When the hdfs-append-enabled() option is set to true, AxoSyslog will not assign a new UUID suffix to an existing file, because it is then possible to open a closed file and append data to that.
Note
You cannot set when log messages are flushed. Hadoop performs this action automatically, depending on its configured block size, and the amount of data received. There is no way for the AxoSyslog application to influence when the messages are actually written to disk. This means that AxoSyslog cannot guarantee that a message sent to HDFS is actually written to disk. When using flow-control, AxoSyslog acknowledges a message as written to disk when it passes the message to the HDFS client. This method is as reliable as your HDFS environment.
If the HDFS client returns an error, AxoSyslog attempts to close the file, then opens a new file and repeats sending the message (trying to connect to HDFS and send the message), as set in the retries() parameter. If sending the message fails for retries() times, AxoSyslog drops the message.
The AxoSyslog application closes the destination file in the following cases:
AxoSyslog is reloaded
AxoSyslog is restarted
The HDFS client returns an error.
If the file is closed and you have set an archive directory, AxoSyslog moves the file to this directory. If AxoSyslog cannot move the file for some reason (for example, AxoSyslog cannot connect to the HDFS NameNode), the file remains at its original location, AxoSyslog will not try to move it again.
8.11.3 - Storing messages with MapR-FS
The AxoSyslog application is also compatible with MapR File System (MapR-FS). MapR-FS provides better performance, reliability, efficiency, maintainability, and ease of use compared to the default Hadoop Distributed Files System (HDFS). To use MapR-FS with AxoSyslog, complete the following steps:
Install MapR libraries. Instead of the official Apache HDFS libraries, MapR uses different libraries. The supported version is MapR 4.x.
Download the libraries from the Maven Repository and Artifacts for MapR or get it from an already existing MapR installation.
Install MapR. If you do not know how to install MapR, follow the instructions on the MapR website.
In a default MapR installation, the required libraries are installed in the following path: /opt/mapr/lib.
Enter the path where MapR was installed in the class-path option of the hdfs destination, for example:
class-path("/opt/mapr/lib/")
If the libraries were downloaded from the Maven Repository, the following additional libraries will be requiered. Note that the version numbers in the filenames can be different in the various Hadoop releases:commons-collections-3.2.1.jar, commons-logging-1.1.3.jar, hadoop-auth-2.5.1.jar, log4j-1.2.15.jar, slf4j-api-1.7.5.jar, commons-configuration-1.6.jar, guava-13.0.1.jar, hadoop-common-2.5.1.jar, maprfs-4.0.2-mapr.jar, slf4j-log4j12-1.7.5.jar, commons-lang-2.5.jar, hadoop-0.20.2-dev-core.jar, json-20080701.jar, protobuf-java-2.5.0.jar, zookeeper-3.4.5-mapr-1406.jar.
Configure the hdfs destination in AxoSyslog.
Example: Storing logfiles with MapR-FS
The following example defines an hdfs destination for MapR-FS using only the required parameters.
8.11.4 - Kerberos authentication with the hdfs() destination
Version 3.10 and later supports Kerberos authentication to authenticate the connection to your Hadoop cluster. AxoSyslog assumes that you already have a Hadoop and Kerberos infrastructure.
Note
If you configure Kerberos authentication for a hdfs() destination, it affects all hdfs() destinations. Kerberos and non-Kerberos hdfs() destinations cannot be mixed in a AxoSyslog configuration. This means that if one hdfs() destination uses Kerberos authentication, you have to configure all other hdfs() destinations to use Kerberos authentication too.
Failing to do so results in non-Kerberos hdfs() destinations being unable to authenticate to the HDFS server.
Note
If you want to configure your hdfs() destination to stop using Kerberos authentication, namely, to remove Kerberos-related options from the hdfs() destination configuration, make sure to restart AxoSyslog for the changes to take effect.
Prerequisites:
You have configured your Hadoop infrastructure to use Kerberos authentication.
You have a keytab file and a principal for the host running AxoSyslog. For details, see the Kerberos documentation.
You have installed and configured the Kerberos client packages on the host running AxoSyslog. (That is, Kerberos authentication works for the host, for example, from the command line using the kinit user@REALM -k -t <keytab_file> command.)
The hdfs destination stores the log messages in files on the Hadoop Distributed File System (HDFS). The hdfs destination has the following options.
The following options are required: hdfs-file(), hdfs-uri(). Note that to use hdfs, you must add the following line to the beginning of your AxoSyslog configuration:
@include "scl.conf"
client-lib-dir()
Type:
string
Default:
The AxoSyslog module directory: /opt/syslog-ng/lib/syslog-ng/java-modules/
Description: The list of the paths where the required Java classes are located. For example, class-path("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/my-java-libraries/libs/"). If you set this option multiple times in your AxoSyslog configuration (for example, because you have multiple Java-based destinations), AxoSyslog will merge every available paths to a single list.
For the hdfs destination, include the path to the directory where you copied the required libraries (see Prerequisites), for example, client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/hadoop/libs/").
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hdfs-append-enabled()
Type:
`true
Default:
false
Description: When hdfs-append-enabled is set to true, AxoSyslog will append new data to the end of an already existing HDFS file. Note that in this case, archiving is automatically disabled, and AxoSyslog will ignore the hdfs-archive-dir option.
When hdfs-append-enabled is set to false, the AxoSyslog application always creates a new file if the previous has been closed. In that case, appending data to existing files is not supported.
When you choose to write data into an existing file, AxoSyslog does not extend the filename with a UUID suffix because there is no need to open a new file (a new unique ID would mean opening a new file and writing data into that).
Warning
Before enabling the hdfs-append-enabled option, ensure that your HDFS server supports the append operation and that it is enabled. Otherwise AxoSyslog will not be able to append data into an existing file, resulting in an error log.
hdfs-archive-dir()
Type:
string
Default:
N/A
Description: The path where AxoSyslog will move the closed log files. If AxoSyslog cannot move the file for some reason (for example, AxoSyslog cannot connect to the HDFS NameNode), the file remains at its original location. For example, hdfs-archive-dir("/usr/hdfs/archive/").
Note
When hdfs-append-enabled is set to true, archiving is automatically disabled, and AxoSyslog will ignore the hdfs-archive-dir option.
hdfs-file()
Type:
string
Default:
N/A
Description: The path and name of the log file. For example, hdfs-file("/usr/hdfs/mylogfile.txt"). AxoSyslog checks if the path to the logfile exists. If a directory does not exist AxoSyslog automatically creates it.
hdfs-file() supports the usage of macros. This means that AxoSyslog can create files on HDFS dynamically, using macros in the file (or directory) name.
Note
When a filename resolved from the macros contains a character that HDFS does not support, AxoSyslog will not be able to create the file. Make sure that you use macros that do not contain unsupported characters.
Example: Using macros in filenames
In the following example, a /var/testdb_working_dir/$DAY-$HOUR.txt file will be created (with a UUID suffix):
As an example, if it is the 31st day of the month and it is 12 o’clock, then the name of the file will be 31-12.txt.
hdfs-max-filename-length()
Type:
number
Default:
255
Description: The maximum length of the filename. This filename (including the UUID that AxoSyslog appends to it) cannot be longer than what the file system permits. If the filename is longer than the value of hdfs-max-filename-length, AxoSyslog will automatically truncate the filename. For example, hdfs-max-filename-length("255").
hdfs-resources()
Type:
string
Default:
N/A
Description: The list of Hadoop resources to load, separated by semicolons. For example, hdfs-resources("/home/user/hadoop/core-site.xml;/home/user/hadoop/hdfs-site.xml").
hdfs-uri()
Type:
string
Default:
N/A
Description: The URI of the HDFS NameNode is in hdfs://IPaddress:port or hdfs://hostname:port format. When using MapR-FS, the URI of the MapR-FS NameNode is in maprfs://IPaddress or maprfs://hostname format, for example: maprfs://10.140.32.80. The IP address of the node can be IPv4 or IPv6. For example, hdfs-uri("hdfs://10.140.32.80:8020"). The IPv6 address must be enclosed in square brackets ([]) as specified by RFC 2732, for example, hdfs-uri("hdfs://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:8020").
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specify the Java Virtual Machine (JVM) settings of your Java destination from the AxoSyslog configuration file.
For example:
jvm-options("-Xss1M -XX:+TraceClassLoading")
You can set this option only as a global option, by adding it to the options statement of the syslog-ng.conf configuration file.
kerberos-keytab-file()
Type:
string
Default:
N/A
Description: The path to the Kerberos keytab file that you received from your Kerberos administrator. For example, kerberos-keytab-file("/opt/syslog-ng/etc/hdfs.headless.keytab"). This option is needed only if you want to authenticate using Kerberos in Hadoop. You also have to set the hdfs-option-kerberos-principal() option. For details on the using Kerberos authentication with the hdfs() destination, see Kerberos authentication with the hdfs() destination.
Description: The Kerberos principal you want to authenticate with. For example, kerberos-principal("hdfs-user@MYREALM"). This option is needed only if you want to authenticate using Kerberos in Hadoop. You also have to set the hdfs-option-kerberos-keytab-file() option. For details on the using Kerberos authentication with the hdfs() destination, see Kerberos authentication with the hdfs() destination.
Description: The number of messages that the output queue can store.
on-error()
Type:
One of: drop-message, drop-property, fallback-to-string, silently-drop-message, silently-drop-property, silently-fallback-to-string
Default:
Use the global setting (which defaults to drop-message)
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reap()
Accepted values:
number (seconds)
Default:
0 (disabled)
Description: The time to wait in seconds before an idle destination file is closed. Note that if hdfs-archive-dir option is set and time-reap expires, archiving is triggered for the affected file.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
8.12 - java: Post messages over HTTP using Java
Version 3.7 of AxoSyslog can directly post log messages to web services using the HTTP protocol. Error and status messages received from the HTTP server are forwarded to the internal logs of AxoSyslog. The current implementation has the following limitations:
Only HTTP connections are supported, HTTPS is not.
If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again.
8.12.1 - HTTP destination options
The http destination of AxoSyslog can directly post log messages to web services using the HTTP protocol. The http destination has the following options. Some of these options are directly used by the Java code underlying the http destination, therefore these options must be specified in the following format:
option("<option-name>", "<option-value>")
For example, option("url", "http://<server-address>:<port-number>"). The exact format to use is indicated in the description of the option.
Required options
The following options are required: url().
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
class-name()
Type:
string
Default:
N/A
Description: The name of the class (including the name of the package) that includes the destination driver to use.
For the http destination, use this option as class-name("org.syslog_ng.http.HTTPDestination").
client-lib-dir()
Type:
string
Default:
The AxoSyslog module directory: /opt/syslog-ng/lib/syslog-ng/java-modules/
Description: The list of the paths where the required Java classes are located. For example, class-path("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/my-java-libraries/libs/"). If you set this option multiple times in your AxoSyslog configuration (for example, because you have multiple Java-based destinations), AxoSyslog will merge every available paths to a single list.
For the http destination, include the path to the java modules of AxoSyslog, for example, class-path("/syslog-ng/install_dir/lib/syslog-ng/java-modules/\*.jar").
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specify the Java Virtual Machine (JVM) settings of your Java destination from the AxoSyslog configuration file.
For example:
jvm-options("-Xss1M -XX:+TraceClassLoading")
You can set this option only as a global option, by adding it to the options statement of the syslog-ng.conf configuration file.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
method()
Type:
DELETE
Default:
PUT
Description: Specifies the HTTP method to use when sending the message to the server. Available in AxoSyslog version 3.7.2 and newer.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
url()
Type:
URL
Default:
Description: Specifies the hostname or IP address and optionally the port number of the web service that can receive log data via HTTP. Use a colon (:) after the address to specify the port number of the server. You can also use macros, templates, and template functions in the URL, for example: http://host.example.com:8080/${MACRO1}/${MACRO2}/script")
8.13 - http: Post messages over HTTP without Java
Version 3.8 of AxoSyslog can directly post log messages to web services using the HTTP protocol, without having to use Java. The current implementation has the following limitations:
Only the PUT and the POST methods are supported.
HTTPS connection, as well as password- and certificate-based authentication is supported.
If the server returns a status code beginning with 2 (for example, 200), AxoSyslog assumes the message was successfully sent. For other response codes, see HTTP destination options. You can override the behavior of AxoSyslog using the response-action() option.
Example: Client certificate authentication with HTTPS
You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.
Alternatively, you can leave the HTTP as-is, in which case the driver leaves the default http_proxy and https_proxy environment variables unmodified.
Starting with version 3.18, you can send multiple log messages in a single HTTP request if the destination HTTP server supports that.
Batch size
The batch-bytes(), batch-lines(), and batch-timeout() options of the destination determine how many log messages AxoSyslog sends in a batch. The batch-lines() option determines the maximum number of messages AxoSyslog puts in a batch in. This can be limited based on size and time:
AxoSyslog sends a batch every batch-timeout() milliseconds, even if the number of messages in the batch is less than batch-lines(). That way the destination receives every message in a timely manner even if suddenly there are no more messages.
AxoSyslog sends the batch if the total size of the messages in the batch reaches batch-bytes() bytes.
To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-bytes(), batch-lines(), batch-timeout() options.
Formatting the batch
By default, AxoSyslog separates the log messages of the batch with a newline character. You can specify a different delimiter by using the delimiter() option.
If the target application or server requires a special beginning or ending to recognize batches, use the body-prefix() and body-suffix() options to add a beginning and ending to the batch. For example, you can use these options to create JSON-encoded arrays as POST payloads, which is required by a number of REST APIs. The body of a batch HTTP request looks like this:
value of body-prefix() option
log-line-1 (as formatted in the body() option) log-line-2 (as formatted in the body() option) ....
log-line-n (the number of log lines is batch-lines(), or less if batch-timeout() has elapsed or the batch would be longer than batch-bytes()) value of body-suffix() option
Starting with version 3.19, you can specify multiple URLs, for example, url("site1" "site2"). In this case, AxoSyslog sends log messages to the specified URLs in a load-balance fashion. This means that AxoSyslog sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order AxoSyslog has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.
Warning
If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.
Starting with version AxoSyslog version 3.22, you can use any of the following formats to specify multiple URLs:
url("server1", "server2", "server3");# comma-separated strings url("server1""server2""server3");# space-separated strings url("server1 server2 server3");# space-separated within a single string
Example: HTTP load balancing
The following destination sends log messages to an Elasticsearch server using the bulk API, to 3 different ingest nodes. Each node is assigned a separate worker thread. A batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 10 seconds (10000 milliseconds).
If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1", "site2", "site3")), set the workers() option to 3 or more.
Templates in the url()
Available in AxoSyslog version 4.5.0 and later.
In AxoSyslog, a template can only be resolved on a single message, because the same template might have different resolutions on different messages. As a batch consists of multiple messages, it’s not trivial to decide which message should be used for the resolution.
When batching is enabled and multiple workers are configured, it’s important to add only those messages to a batch which generate identical URLs. To achieve this, set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed.
For security reasons, all the templated contents in the url() option are URL-encoded automatically. The following parts of the URL cannot be templated:
scheme
host
port
user
password
8.13.2 - HTTP destination options
The http destination of AxoSyslog can directly post log messages to web services using the HTTP protocol. The http destination has the following options.
accept-encoding()
Type:
"identity", "gzip", "deflate", "all"
Default:
Description: Use accept-encoding() to request the server to compress the HTTP responses. (AxoSyslog doesn’t currently use them, but they still contribute to network traffic.) To compress the messages sent by AxoSyslog, see the content-compression() option.
Use "identity" for no compression.
If you want to accept multiple compression types, list them separated by commas inside the quotation mark.
To enable all available compression types, use "all".
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
Description: The body of the HTTP request, for example, body("${ISODATE} ${MESSAGE}"). You can use strings, macros, and template functions in the body. If not set, it will contain the message received from the source by default.
body-prefix()
Accepted values:
string
Default:
none
Description: The string AxoSyslog puts at the beginning of the body of the HTTP request, before the log message. Available in AxoSyslog version 3.18 and later.
Description: The string AxoSyslog puts to the end of the body of the HTTP request, after the log message. Available in AxoSyslog version 3.18 and later.
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Name of a file that contains an X.509 CA certificate (or a certificate chain) in PEM format. The AxoSyslog application uses this certificate to validate the certificate of the HTTPS server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Name of a file, that contains an X.509 certificate (or a certificate chain) in PEM format, suitable as a TLS certificate, matching the private key set in the key-file() option. The AxoSyslog application uses this certificate to authenticate the AxoSyslog client on the destination server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Depends on the OpenSSL version that AxoSyslog uses
Description: Specifies the cipher, hash, and key-exchange algorithms used for the encryption, for example, ECDHE-ECDSA-AES256-SHA384. The list of available algorithms depends on the version of OpenSSL used to compile AxoSyslog. To specify multiple ciphers, separate the cipher names with a colon, and enclose the list between double-quotes, for example:
For a list of available algorithms, execute the openssl ciphers -v command. The first column of the output contains the name of the algorithms to use in the cipher-suite() option, the second column specifies which encryption protocol uses the algorithm (for example, TLSv1.2). That way, the cipher-suite() also determines the encryption protocol used in the connection: to disable SSLv3, use an algorithm that is available only in TLSv1.2, and that both the client and the server supports. You can also specify the encryption protocols using ssl-options().
You can also use the following command to automatically list only ciphers permitted in a specific encryption protocol, for example, TLSv1.2:
echo"cipher-suite(\"$(openssl ciphers -v | grep TLSv1.2 | awk '{print $1}'| xargs echo -n | sed 's/ /:/g'| sed -e 's/:$//')\")"
Note that starting with version 3.10, when AxoSyslog receives TLS-encrypted connections, the order of ciphers set on the AxoSyslog server takes precedence over the client settings.
content-compression()
Type:
"identity", "gzip", "deflate", "all"
Default:
Description: Use content-compression() to compress the messages sent by AxoSyslog. To accept compressed responses from the server, see the accept-encoding() option.
Use "identity" for no compression.
If you want to accept multiple compression types, list them separated by commas inside the quotation mark.
To enable all available compression types, use "all".
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: By default, AxoSyslog separates the log messages of the batch with a newline character. You can specify a different delimiter by using the delimiter() option. Available in AxoSyslog version 3.18 and later.
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
flush-timeout()
Type:
time in milliseconds
Default:
10000 [milliseconds]
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends flushes to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every flush-timeout() seconds.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
headers()
Type:
string list
Default:
Description: Custom HTTP headers to include in the request, for example, headers("HEADER1: header1", "HEADER2: header2"). If not set, only the default headers are included, but no custom headers.
The following headers are included by default:
X-Syslog-Host: <host>
X-Syslog-Program: <program>
X-Syslog-Facility: <facility>
X-Syslog-Level: <loglevel/priority>
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The name of a file that contains an unencrypted private key in PEM format, suitable as a TLS key. If properly configured, the AxoSyslog application uses this private key and the matching certificate (set in the cert-file() option) to authenticate the AxoSyslog client on the destination server.
The http() destination supports only unencrypted key files (that is, the private key cannot be password-protected).
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Sets the timezone used when expanding filename and tablename templates.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
method()
Type:
POST or PUT
Default:
POST
Description: Specifies the HTTP method to use when sending the message to the server.
ocsp-stapling-verify
Accepted values:
yes, no
Default:
no
Available in AxoSyslog 4.0 and later.
Description: When OCSP stapling verification is enabled, AxoSyslog requests the server to send back its OCSP status. AxoSyslog verifies this status response using the trust store you have configured using the ca-file(), ca-dir(), or the pkcs12-file() options.
Note
RFC 6961 multi-stapling and TLS 1.3-provided multiple responses are currently not validated, only the peer certificate is verified.
One of: drop-message, drop-property, fallback-to-string, silently-drop-message, silently-drop-property, silently-fallback-to-string
Default:
Use the global setting (which defaults to drop-message)
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
password()
Type:
string
Default:
Description: The password that AxoSyslog uses to authenticate on the server where it sends the messages.
peer-verify()
Accepted values:
`yes
Default:
yes
Description: Verification method of the peer. The following table summarizes the possible options and their results depending on the certificate of the peer.
The remote peer has:
no certificate
invalid certificate
valid certificate
Local peer-verify() setting
no (optional-untrusted)
TLS-encryption
TLS-encryption
TLS-encryption
yes (required-trusted)
rejected connection
rejected connection
TLS-encryption
For untrusted certificates only the existence of the certificate is checked, but it does not have to be valid — AxoSyslog accepts the certificate even if it is expired, signed by an unknown CA, or its CN and the name of the machine mismatches.
Warning
When validating a certificate, the entire certificate chain must be valid, including the CA certificate. If any certificate of the chain is invalid, AxoSyslog will reject the connection.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
proxy()
Type:
The proxy server address, in `proxy("PROXY_IP:PORT")` format.
For example, `proxy("http://myproxy:3128")`
Default:
None
Description:
You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.
Alternatively, you can leave the HTTP as-is, in which case the driver leaves the default http_proxy and https_proxy environment variables unmodified.
Note
Configuring the <code>proxy()</code> option overwrites the default <code>http_proxy</code> and <code>https_proxy</code> environment variables.
Example: the proxy() option in configuration
The following example illustrates including the proxy() option in your configuration.
Description: Specifies what AxoSyslog does with the log message, based on the response code received from the HTTP server. If the server returns a status code beginning with 2 (for example, 200), AxoSyslog assumes the message was successfully sent. Otherwise, the action listed in the following table is applied. For status codes not listed in the following table, if the status code begins with 2 (for example, 299), AxoSyslog assumes the message was successfully sent. For other status codes, AxoSyslog disconnects. The following actions are possible:
disconnect: Keep trying to resend the message indefinitely.
drop: Drop the message without trying to resend it.
retry: Retry sending the message for a maximum of retries() times (3 by default).
success: Assume the message was successfully sent.
|------+-----------------------------------+------------|| code | explanation | action ||------+-----------------------------------+------------||100|"Continue"| disconnect ||101|"Switching Protocols"| disconnect ||102|"Processing"| retry ||103|"Early Hints"| retry ||200|"OK"| success ||201|"Created"| success ||202|"Accepted"| success ||203|"Non-Authoritative Information"| success ||204|"No Content"| success ||205|"Reset Content"| success ||206|"Partial Content"| success ||300|"Multiple Choices"| disconnect ||301|"Moved Permanently"| disconnect ||302|"Found"| disconnect ||303|"See Other"| disconnect ||304|"Not Modified"| retry ||307|"Temporary Redirect"| disconnect ||308|"Permanent Redirect"| disconnect ||400|"Bad Request"| disconnect ||401|"Unauthorized"| disconnect ||402|"Payment Required"| disconnect ||403|"Forbidden"| disconnect ||404|"Not Found"| disconnect ||405|"Method Not Allowed"| disconnect ||406|"Not Acceptable"| disconnect ||407|"Proxy Authentication Required"| disconnect ||408|"Request Timeout"| disconnect ||409|"Conflict"| disconnect ||410|"Gone"| drop ||411|"Length Required"| disconnect ||412|"Precondition Failed"| disconnect ||413|"Payload Too Large"| disconnect ||414|"URI Too Long"| disconnect ||415|"Unsupported Media Type"| disconnect ||416|"Range Not Satisfiable"| drop ||417|"Expectation Failed"| disconnect ||418|"I'm a teapot"| disconnect ||421|"Misdirected Request"| disconnect ||422|"Unprocessable Entity"| drop ||423|"Locked"| disconnect ||424|"Failed Dependency"| drop ||425|"Too Early"| drop ||426|"Upgrade Required"| disconnect ||428|"Precondition Required"| retry ||429|"Too Many Requests"| disconnect ||431|"Request Header Fields Too Large"| disconnect ||451|"Unavailable For Legal Reasons"| drop ||500|"Internal Server Error"| disconnect ||501|"Not Implemented"| disconnect ||502|"Bad Gateway"| disconnect ||503|"Service Unavailable"| disconnect ||504|"Gateway Timeout"| retry ||505|"HTTP Version Not Supported"| disconnect ||506|"Variant Also Negotiates"| disconnect ||507|"Insufficient Storage"| disconnect ||508|"Loop Detected"| drop ||510|"Not Extended"| disconnect ||511|"Network Authentication Required"| disconnect ||------+-----------------------------------+------------|
To customize the action to take for a particular response code, use the following format: response-action(<response-code> => <action>. To customize multiple response code-action pairs, separate them with a comma, for example:
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
To handle HTTP error responses, if the HTTP server returns 5xx codes, AxoSyslog will attempt to resend messages until the number of attempts reaches retries. If the HTTP server returns 4xx codes, AxoSyslog will drop the messages.
send-time-zone()
Accepted values:
name of the timezone, or the timezone offset
Default:
local timezone
Description: Specifies the time zone associated with the messages sent by syslog-ng, if not specified otherwise in the message or in the destination driver. For details, see Timezones and daylight saving.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ssl-version()
Type:
string
Default:
None, uses the libcurl default
Description: Specifies the permitted SSL/TLS version. Possible values: sslv2, sslv3, tlsv1, tlsv1_0, tlsv1_1, tlsv1_2, tlsv1_3.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
timeout()
Type:
number [seconds]
Default:
0
Description: The value (in seconds) to wait for an operation to complete, and attempt to reconnect the server if exceeded. By default, the timeout value is 0, meaning that there is no timeout. Available in version 3.11 and later.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
url()
Type:
URL or list of URLs
Default:
http://localhost/
Description: Specifies the hostname or IP address and optionally the port number of the web service that can receive log data via HTTP. Use a colon (:) after the address to specify the port number of the server. For example: http://127.0.0.1:8000
In case the server on the specified URL returns a redirect request, AxoSyslog automatically follows maximum 3 redirects. Only HTTP and HTTPS based redirections are supported.
Templates in the URL
Starting with AxoSyslog version 4.5, you can use templates in the url() option of the http() driver, but note the following points.
Templates and batching: A template can only be resolved for a single message, because the template might have different resolutions on different messages. When batching is enabled and the destination uses multiple workers, it’s important to only batch messages which generate identical URLs. In this case, set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed.
For security reasons, AxoSyslog automatically URL-encodes the templated content of the url() option. The following parts of the URL cannot be templated:
scheme
host
port
user
password
Load balancing
Starting with version 3.19, you can specify multiple URLs, for example, url("site1" "site2"). In this case, AxoSyslog sends log messages to the specified URLs in a load-balance fashion. This means that AxoSyslog sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order AxoSyslog has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.
Warning
If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.
Starting with version AxoSyslog version 3.22, you can use any of the following formats to specify multiple URLs:
url("server1", "server2", "server3");# comma-separated strings url("server1""server2""server3");# space-separated strings url("server1 server2 server3");# space-separated within a single string
Templates in the url()
Available in AxoSyslog version 4.5.0 and later.
In AxoSyslog, a template can only be resolved on a single message, because the same template might have different resolutions on different messages. As a batch consists of multiple messages, it’s not trivial to decide which message should be used for the resolution.
When batching is enabled and multiple workers are configured, it’s important to add only those messages to a batch which generate identical URLs. To achieve this, set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed.
For security reasons, all the templated contents in the url() option are URL-encoded automatically. The following parts of the URL cannot be templated:
scheme
host
port
user
password
user-agent()
Type:
string
Default:
syslog-ng [version]'/libcurl[version]`
Description: The value of the USER-AGENT header in the messages sent to the server.
user()
Type:
string
Default:
Description: The username that AxoSyslog uses to authenticate on the server where it sends the messages.
use-system-cert-store()
Type:
`yes
Default:
no
Description: Use the certificate store of the system for verifying HTTPS certificates. For details, see the curl documentation.
worker-partition-key()
Type:
template
Default:
Description: The worker-partition-key() option specifies a template: messages that expand the template to the same value are mapped to the same partition. When batching is enabled and multiple workers are configured, it’s important to add only those messages to a batch which generate identical URLs. To achieve this, set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed.
For example, you can partition messages based on the destination host:
worker-partition-key("$HOST")
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1", "site2", "site3")), set the workers() option to 3 or more.
8.13.3 - The Azure auth header plugin
This section describes the AxoSyslog application’s Azure auth header plugin.
For more information about modules in AxoSyslog, see Using modules.
The Azure auth header plugin
The Azure auth header plugin is a signal-slot mechanism-based AxoSyslog module that generates authorization headers for applications that connect to Microsoft Azure.
Defining the Azure auth header plugin
You can define the Azure auth header plugin by the following:
All these options are mandatory. They are used as input for the method that calculates the authorization header.
8.13.4 - The Python HTTP header plugin
This section describes the AxoSyslog application’s Python HTTP header plugin.
For more information about modules in AxoSyslog, see Using modules.
The Python HTTP header plugin
The AxoSyslog application supports adding custom headers to HTTP requests using the Python programming language.
Prerequisites
Note
Before you use the python-http-header plugin, make sure that your AxoSyslog appliance was compiled with Python support. If you installed AxoSyslog from a package, make sure that the subpackage containing Python support is also installed.
class: Mandatory option. It refers to the user’s Python class that implements the python-http-header interface. It can be mymodule.MyClass if the class MyClass is put into a mymodule.py module, or simply MyClass if the user’s code is provided inline in the configuration, using the python { ... }; keyword.
Note
If you put the class implementation into its own module, it should be put into a standard location, or made available with the PYTHONPATH environment variable.
options("key" "value"): Optional option. Multiple options can be specified at the same time. The AxoSyslog application will build a Python dictionary, which will be available in the __init__ method.
mark-errors-as-critical(yes|no): Optional option. Its default value is yes. In case there is a Python error, this parameter decides if the HTTP destination will still try to send the request with the failed headers, or disconnect instead.
Defining the python-http-header() interface
You can define the Python interface with the following:
By default, when the signal_http_header_request is emitted by the HTTP module, the connected slot automatically executes the Python code.
Note
If the plugin fails, the HTTP module does not send the HTTP request without the header items by default. If you want the HTTP module to try sending the request without the header items, disable the mark-errors-as-critical function.
Methods used in the configuration:
__init__(self, options): Optional method. The options specified in the AxoSyslog configuration can be stored in the instance using this method.
get_headers(self, body, headers): Mandatory method. Returns a list of strings of form ["header: value", …]. The returned headers will be set for the outgoing HTTP request. The body contains the body of the HTTP request. The headers contain the current headers that the HTTP destination has already added to the request.
on_http_response_received(self, http_code): Optional method. If specified, AxoSyslog inserts the http_code of the previous response. This can be used to handle error (for example, for recreating auth headers, or dropping cache).
Example configuration for using the Python HTTP header plugin
The following example can be copy-pasted and used as a template for using the Python HTTP header plugin in your configuration.
python { from syslogng import Logger
logger= Logger() class TestCounter():
def __init__(self, options):
self.header = options["header"] self.counter = int(options["counter"]) logger.debug(f"TestCounter class instantiated; options={options}") def get_headers(self, body, headers):
logger.debug(f"get_headers() called, received body={body}, headers={headers}")response=["{}: {}".format(self.header, self.counter)] self.counter +=1return response
def on_http_response_received(self, http_code):
self.counter += http_code
logger.debug("HTTP response code received: {}".format(http_code)) def __del__(self):
logger.debug("Deleting TestCounter class instance")};source s_network { network(port(5555));}; destination d_http { http( python_http_header( class("TestCounter") options("header", "X-Test-Python-Counter") options("counter", 11)# this means that syslog-ng will keep trying to send the http request even when this module fails mark-errors-as-critical(no)) url("http://127.0.0.1:8888"));}; log { source(s_network); destination(d_http); flags(flow-control);};
Warning
Although it is possible to configure multiple HTTP workers for AxoSyslog, the AxoSyslog application can only embed a single Python interpreter at the same time. As a result, if you configure more than one HTTP workers on your AxoSyslog application, the Python code will run in concurrent mode. To protect the state of the object, you may need to use locks.
The kafka() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of AxoSyslog. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the kafka configuration snippet on GitHub. For details on extending AxoSyslog in Java, see the Getting started with syslog-ng development guide.
Note
If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again.
8.14.1 - Prerequisites
To publish messages from AxoSyslog to Apache Kafka, complete the following steps.
Steps:
If you want to use the Java-based modules of AxoSyslog (for example, the Elasticsearch, HDFS, or Kafka destinations), you must compile AxoSyslog with Java support.
Download and install the Java Runtime Environment (JRE), 1.7 (or newer). You can use OpenJDK or Oracle JDK, other implementations are not tested.
Set LD_LIBRARY_PATH to include the libjvm.so file, for example:LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH
Note that many platforms have a simplified links for Java libraries. Use the simplified path if available. If you use a startup script to start AxoSyslog set LD_LIBRARY_PATH in the script as well.
If you are behind an HTTP proxy, create a gradle.properties under the modules/java-modules/ directory. Set the proxy parameters in the file. For details, see The Gradle User Guide.
Extract the Apache Kafka libraries into a single directory. If needed, collect the various .jar files into a single directory (for example, /opt/kafka/lib/) where AxoSyslog can access them. You must specify this directory in the AxoSyslog configuration file.
Check if the following files in the Kafka libraries have the same version number: slf4j-api-<version-number>.jar, slf4j-log4j12-<version-number>.jar. If the version number of these files is different, complete the following steps:
Delete one of the files (for example, slf4j-log4j12-<version-number>.jar).
Download a version that matches the version number of the other file (for example, 1.7.6) from the official SLF4J distribution.
Copy the downloaded file into the directory of your Kafka library files (for example, /opt/kafka/lib/).
8.14.2 - How AxoSyslog interacts with Apache Kafka
When stopping the AxoSyslog application, AxoSyslog will not stop until all Java threads are finished, including the threads started by the Kafka Producer. There is no way (except for the kill -9 command) to stop AxoSyslog before the Kafka Producer stops. To change this behavior set the properties of the Kafka Producer in its properties file, and reference the file in the properties-file option.
The AxoSyslog kafka destination tries to reconnect to the brokers in a tight loop. This can look as spinning, because of a lot of similar debug messages. To decrease the amount of such messages, set a bigger timeout using the following properties:
The kafka destination of AxoSyslog can directly publish log messages to the Apache Kafka message bus, where subscribers can access them. The kafka destination has the following options.
Required options:
The following options are required: kafka-bootstrap-servers(), topic(). Note that to use kafka, you must add the following lines to the beginning of your AxoSyslog configuration:
@include "scl.conf"
client-lib-dir()
Type:
string
Default:
The AxoSyslog module directory: /opt/syslog-ng/lib/syslog-ng/java-modules/
Description: The list of the paths where the required Java classes are located. For example, class-path("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/my-java-libraries/libs/"). If you set this option multiple times in your AxoSyslog configuration (for example, because you have multiple Java-based destinations), AxoSyslog will merge every available paths to a single list.
For the kafka destination, include the path to the directory where you copied the required libraries (see Prerequisites), for example, client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/\*.jar").
kafka-bootstrap-servers()
Type:
list of hostnames
Default:
Description: Specifies the hostname or IP address of the Kafka server. When specifying an IP address, IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well. Use a colon (:) after the address to specify the port number of the server. When specifying multiple addresses, use a comma to separate the addresses, for example, kafka-bootstrap-servers("127.0.0.1:2525,remote-server-hostname:6464")
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specify the Java Virtual Machine (JVM) settings of your Java destination from the AxoSyslog configuration file.
For example:
jvm-options("-Xss1M -XX:+TraceClassLoading")
You can set this option only as a global option, by adding it to the options statement of the syslog-ng.conf configuration file.
on-error()
Type:
One of: drop-message, drop-property, fallback-to-string, silently-drop-message, silently-drop-property, silently-fallback-to-string
Default:
Use the global setting (which defaults to drop-message)
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
key()
Type:
template
Default:
N/A
Description: The key of the partition under which the message is published. You can use templates to change the topic dynamically based on the source or the content of the message, for example, key("${PROGRAM}").
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
properties-file()
Type:
string (absolute path)
Default:
N/A
Description: The absolute path and filename of the Kafka properties file to load. For example, properties-file("/opt/syslog-ng/etc/kafka_dest.properties"). The AxoSyslog application reads this file and passes the properties to the Kafka Producer. If a property is defined both in the AxoSyslog configuration file (syslog-ng.conf) and in the properties file, then AxoSyslog uses the definition from the AxoSyslog configuration file.
The AxoSyslog kafka destination supports all properties of the official Kafka producer. For details, see the Apache Kafka documentation.
The kafka-bootstrap-servers option is translated to the bootstrap.servers property.
For example, the following properties file defines the acknowledgment method and compression:
acks=all
compression.type=snappy
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
sync-send()
Type:
`true
Default:
false
Description: When sync-send is set to true, AxoSyslog sends the message reliably: it sends a message to the Kafka server, then waits for a reply. In case of failure, AxoSyslog repeats sending the message, as set in the retries() parameter. If sending the message fails for retries() times, AxoSyslog drops the message.
This method ensures reliable message transfer, but is very slow.
When sync-send() is set to false, AxoSyslog sends messages asynchronously, and receives the response asynchronously. In case of a problem, AxoSyslog cannot resend the messages.
This method is fast, but the transfer is not reliable. Several thousands of messages can be lost before AxoSyslog recognizes the error.
template()
Type:
template or template function
Default:
$ISODATE $HOST $MSGHDR$MSG\\n
Description: The message as published to Apache Kafka. You can use templates and template functions (for example, format-json()) to format the message, for example, template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE)").
For details on formatting messages in JSON format, see format-json.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
topic()
Type:
template
Default:
N/A
Description: The Kafka topic under which the message is published. You can use templates to change the topic dynamically based on the source or the content of the message, for example, topic("${HOST}").
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
8.15 - kafka-c(): Publish messages to Apache Kafka (C implementation)
Starting with version 3.21, AxoSyslog can directly publish log messages to the Apache Kafka message bus, where subscribers can access them.
As of AxoSyslog version 3.21, the new C implementation of the kafka destination is available. The new implementation uses the librdkafka client and has several advantages, such as scalability, more efficient memory usage and simpler setup. The options of this implementation are compatible with those of the old Java implementation.
8.15.1 - Shifting from Java implementation to C implementation
If you were using the Java implementation of the kafka destination and want to shift to its C implementation, the following changes to the configuration file and considerations are necessary.
Unlike the old one, the new topic() option can not handle templates. It must be a string.
The template() option has been renamed message().
The kafka-bootstrap-servers() option has been renamed bootstrap-servers().
The properties-file() is a Java properties file with options that are similar to, but not identical with, the options in the old, Java implementation’s properties-file(). For more information, click here.
The sync-send() option has been deprecated. Remove it from the configuration file.
The client_lib_dir() option has been deprecated. Remove it from the configuration file.
The old implementation’s option() option has been removed and replaced by the config() option, which has a different syntax.
This section describes the prerequisites and restrictions for using the kafka destination in the new C implementation, and important information about the declaration of the destination.
Prerequisites and restrictions
Since the new C implementation uses the librdkafka client library, the kafka destination has less memory usage than the previous Java implementation (which uses the official Java Kafka producer).
A AxoSyslog destination recognizes a message as sent when the message has been sent to the Kafka client, not when the Kafka server confirms its delivery.
If the Kafka client collects too many unsent messages, it will not accept any more messages from AxoSyslog. The AxoSyslog application detects this and stops sending messages to the Kafka client. Also, AxoSyslog’s flow control starts functioning in the direction of the sources (for example, AxoSyslog will not read from the sources in that specific logpath).
You can specify a “high water mark” limit for the Kafka client in the properties-file().
For more information about how the C implementation of the kafka() destination works with AxoSyslog, click here.
8.15.4 - Options of the kafka() destination's C implementation
The C implementation of the kafka() destination of AxoSyslog can directly publish log messages to the Apache Kafka message bus, where subscribers can access them. The C implementation of the kafka() destination has the following options.
Required options:
The following options are required: bootstrap-servers(), topic(). Note that to use the C implementation of the kafka() destination, you must add the following lines to the beginning of your AxoSyslog configuration:
@define kafka-implementation kafka-c
Note
At least one of the config() and the properties_file() options is mandatory. While you can specify everything in the config() option if you want, the properties-file() is optional. If you have an option in both the config() and the properties-file() specified, the option specified later in the AxoSyslog configuration file will prevail.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
Note
The AxoSyslog configuration accepts batch-lines() with sync-send() set to both "yes" or "no", but the option will only take effect if you set sync-send() to "yes".
Note
If you set sync-send() to "yes", the number you specify for batch-lines() affects how many messages AxoSyslog packs into once transaction.
Note
When setting batch-timeout(), consider the value of the transaction.timeout.ms Kafka property. If in case of timeout (that is, if AxoSyslog does not receive batch-lines() amount of messages) the value of batch-timeout() exceeds the value of transaction.timeout.ms, AxoSyslog will not send out messages in time.
For more information about the default values of the transaction.timeout.ms Kafka property, see the librdkafka documentation.
bootstrap-servers()
Type:
string
Default:
Description: Specifies the hostname or IP address of the Kafka server. When specifying an IP address, IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well. Use a colon (:) after the address to specify the port number of the server. When specifying multiple addresses, use a comma to separate the addresses, for example, bootstrap-servers("127.0.0.1:2525,remote-server-hostname:6464")
client-lib-dir()
Type:
string
Default:
The AxoSyslog module directory: /opt/syslog-ng/lib/syslog-ng/java-modules/
Description: The list of the paths where the required Java classes are located. For example, class-path("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/my-java-libraries/libs/"). If you set this option multiple times in your AxoSyslog configuration (for example, because you have multiple Java-based destinations), AxoSyslog will merge every available paths to a single list.
For the kafka destination, include the path to the directory where you copied the required libraries (see Prerequisites), for example, client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/\*.jar").
Note
Unlike in the Java implementation, the client-lib-dir() option has no significant role in the C implementation of the kafka() destination. The programming language accepts this option for better compatibility.
config()
Description: You can use this option to expand or override the options of the properties-file().
Note
At least one of the config() and the properties_file() options is mandatory. While you can specify everything in the config() option if you want, the properties-file() is optional. If you have an option in both the config() and the properties-file() specified, the option specified later in the AxoSyslog configuration file will prevail.
The AxoSyslogkafka destination supports all properties of the official Kafka producer. For details, see the librdkafka documentation.
The syntax of the config() option is the following:
config( “key1” => “value1”
“key2” => “value2”
)
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
flush-timeout-on-reload()
Type:
integer in msec
Default:
1000
Description: When AxoSyslog reloads, the Kafka client will also reload. The flush-timeout-on-reload() option specifies the number of milliseconds AxoSyslog waits for the Kafka client to send the unsent messages. The unsent messages will be retained in syslog-ng’s own queue and AxoSyslog will continue sending them after reload. This works without disk-buffering, too.
flush-timeout-on-shutdown()
Type:
integer in msec
Default:
60000
Description: When AxoSyslog shuts down, the Kafka client will also shut down. The flush-timeout-on-shutdown() option specifies the number of milliseconds AxoSyslog waits for the Kafka client to send the unsent messages. Any messages not sent after the specified time will be lost. To avoid losing messages, we recommend you use the disk-buffer option.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The key of the partition under which the message is published. You can use templates to change the topic dynamically based on the source or the content of the message, for example, key("${PROGRAM}").
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
local-time-zone()
Type:
name of the timezone, or the timezone offset
Default:
The local timezone.
Description: Sets the timezone used when expanding filename and tablename templates.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
on-error()
Type:
One of: drop-message, drop-property, fallback-to-string, silently-drop-message, silently-drop-property, silently-fallback-to-string
Default:
Use the global setting (which defaults to drop-message)
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
poll-timeout()
Type:
integer in msec
Default:
1000
Description: Specifies the frequency your AxoSyslog queries the Kafka client about the amount of messages sent since the last poll-timeout (). In case of multithreading, the first AxoSyslog worker is responsible for poll-timeout().
properties-file()
Type:
string (absolute path)
Default:
N/A
Description: The absolute path and filename of the Kafka properties file to load. For example, properties-file("/opt/syslog-ng/etc/kafka_dest.properties"). The AxoSyslog application reads this file and passes the properties to the Kafka Producer.
The AxoSyslogkafka destination supports all properties of the official Kafka producer. For details, see the librdkafka documentation.
The bootstrap-servers option is translated to the bootstrap.servers property.
For example, the following properties file defines the acknowledgment method and compression:
example
`acks=all
compression.type=snappy`.
Note
At least one of the config() and the properties_file() options is mandatory. While you can specify everything in the config() option if you want, the properties-file() is optional. If you have an option in both the config() and the properties-file() specified, the option specified later in the AxoSyslog configuration file will prevail.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
send-time-zone()
Accepted values:
name of the timezone, or the timezone offset
Default:
local timezone
Description: Specifies the time zone associated with the messages sent by syslog-ng, if not specified otherwise in the message or in the destination driver. For details, see Timezones and daylight saving.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
sync-send()
Type:
`true
Default:
false
Description: When sync-send is set to true, AxoSyslog sends the message reliably: it sends a message to the Kafka server, then waits for a reply. In case of failure, AxoSyslog repeats sending the message, as set in the retries() parameter. If sending the message fails for retries() times, AxoSyslog drops the message.
This method ensures reliable message transfer, but is very slow.
When sync-send() is set to false, AxoSyslog sends messages asynchronously, and receives the response asynchronously. In case of a problem, AxoSyslog cannot resend the messages.
This method is fast, but the transfer is not reliable. Several thousands of messages can be lost before AxoSyslog recognizes the error.
Note
If you want to use the sync-send() option set to "yes", Axoflow recommends that you use librdkafka version 1.4.0 or higher, and a Kafka server with version number 0.11.0 or higher.
template()
Type:
template or template function
Default:
$ISODATE $HOST $MSGHDR$MSG\\n
Description: The message as published to Apache Kafka. You can use templates and template functions (for example, format-json()) to format the message, for example, template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE)").
For details on formatting messages in JSON format, see format-json.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
topic()
Type:
string
Default:
N/A
Description: The Kafka topic under which the message is published.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
Note
The workers are only responsible for formatting the messages that need to be delivered to the Kafka clients. Configure this option only if your Kafka clients have many threads and they do not receive enough messages.
Note
Kafka clients have their own threadpool, entirely independent from any AxoSyslog settings. The workers() option has no effect on this threadpool.
8.16 - loggly: Send logs to Loggly
The loggly() destination sends log messages to the Loggly Logging-as-a-Service provider. You can send log messages over TCP, or encrypted with TLS.
Declaration:
loggly(token());
Example: Using the loggly() driver
To use the loggly() destination, the only mandatory parameter is your user token. The following example sends every log from the system() source to your Loggly account.
The following example uses TLS encryption. Before using it, download the CA certificate of Loggly and copy it to your hosts (for example, into the /etc/ssl/certs/ directory.
To use the loggly() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The loggly() driver is actually a reusable configuration snippet configured to send log messages using the tcp() driver using a template. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
8.16.1 - loggly() destination options
The loggly() destination has the following options. You can also set other options of the underlying tcp() driver (for example, port number or TLS-encryption).
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
token()
Type:
string
Default:
Description: Your Customer Token that you received from Loggly.
transport()
Type:
udp, tcp, or tls
Default:
tcp
Description: Specifies the protocol used to send messages to the destination server.
If you use the udp transport, AxoSyslog automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.
8.17 - logmatic: Send logs to Logmatic.io
The logmatic() destination sends log messages to the Logmatic.io Logging-as-a-Service provider. You can send log messages over TCP, or encrypted with TLS.
Declaration:
logmatic(token());
Example: Using the logmatic() driver
To use the logmatic() destination, the only mandatory parameter is your user token. The following example sends every log from the system() source to your Logmatic.io account.
The following example uses TLS encryption. Before using it, download the CA certificate of Logmatic.io and copy it to your hosts (for example, into the /etc/ssl/certs/ directory.
To use the logmatic() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The logmatic() driver is actually a reusable configuration snippet configured to send log messages using the tcp() driver using a template. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
8.17.1 - logmatic() destination options
The logmatic() destination has the following options. You can also set other options of the underlying tcp() driver (for example, port number or TLS-encryption).
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Your API Key that you received from Logmatic.io.
8.18 - Send messages to Falcon LogScale
Starting with version 4.3.0, AxoSyslog can send messages to Falcon LogScale using its Ingest Structured Data API. That way you don’t have to parse the data on Falcon LogScale, because AxoSyslog already sends it in a structured format that LogScale understands and can show in a structured manner as separate columns. For a tutorial on using this destination in Kubernetes, see the From syslog-ng to LogScale: structured logs from any source blog post.
Prerequisites
Create an Ingest token for AxoSyslog to use in the token() option of the destination. This token is specific to a LogScale repository.
This driver is actually a reusable configuration snippet configured to send log messages using the http() driver using a template. You can find the source of this configuration snippet on GitHub.
Options
The following options are specific to the logscale() destination. But since this destination is based on the http() destination, you can use the options of the http() destination as well if needed.
attributes()
Type:
string
Default:
"--scope rfc5424 --exclude MESSAGE --exclude DATE --leave-initial-dot"
Description: A JSON object representing key-value pairs for the LogScale Event, formatted as AxoSyslog value-pairs. By default, the logscale() destination sends the RFC5424 fields as attributes. If you want to send different fields, override the default template.
content-type()
Type:
string
Default:
"application/json"
Description: The content-type of the HTTP request.
extra-headers()
Type:
string
Default:
Description: Extra headers for the HTTP request.
rawstring()
Type:
template
Default:
${MESSAGE}
Description: Accepts a template that you can use to format the LogScale event.
Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.
tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.
batch-lines()
Type:
number
Default:
0
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
channel-args()
Type:
See description
Default:
-
Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:
Description: The maximum number of gRPC pings that can be sent when there is no data/header frame to be sent. AxoSyslog won’t send any pings after this limit. Set it to 0 disable this restriction and keep sending pings.
time()
Type:
number [milliseconds]
Default:
Description: The period (in milliseconds) after which AxoSyslog sends a gRPC keepalive ping.
timeout()
Type:
number [milliseconds]
Default:
10
Description: The time (in milliseconds) AxoSyslog waits for an acknowledgement.
labels()
Type:
Default:
See the description
The labels applied to the message as they are sent to the destination. Use the following format:
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. For details on template functions, see Template functions of AxoSyslog.
tenant-id()
Type:
string
Default:
-
Description: Available in version 4.7 and newer. Sets the tenant ID for multi-tenant scenarios. For example:
Description: Sets the timestamp to use for the messages sent to Loki. This is important because Loki accepts data only if their timestamp is monotonously increasing, out of order messages are rejected. The possible values for this option are:
current: Use the timestamp when AxoSyslog processes the message in the output. This guarantees that the timestamp is monotonously increasing, but in some cases can significantly differ from the time when the message was generated.
msg: Use the original timestamp of the message.
received: Use the timestamp when AxoSyslog has received the message.
url()
Type:
string
Default:
localhost:9095
Description: The URL of the Loki endpoint.
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
8.20 - mongodb(): Store messages in a MongoDB database
The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database. For the list of available optional parameters, see mongodb() destination options.
Declaration
mongodb(parameters);
The mongodb() driver does not support creating indexes, as that can be a very complex operation in MongoDB. If needed, the administrator of the MongoDB database must ensure that indexes are created on the collections.
The mongodb() driver does not add the _id field to the message: the MongoDB server will do that automatically, if none is present. If you want to override this field from AxoSyslog, use the key() parameter of the value-pairs() option.
The AxoSyslog mongodb() driver is compatible with MongoDB server version 1.4 and newer.
Note
Prior to version 4.0, AxoSyslog handled every data as strings, and allowed you to convert the strings into other types of data that only certain destinations data formats supported.
Starting with AxoSyslog 4.0, each name-value pair is a (name, type, value) triplet, and several components of AxoSyslog have typing support. For details, see Components that support data types.
Example: Using the mongodb() driver
The following example creates a mongodb() destination using only default values.
destination d_mongodb { mongodb();};
The following example displays the default values.
The following example shows the same setup using the deprecated libmongo-client syntax (as used in AxoSyslog version 3.7), and is equivalent with the previous example.
When AxoSyslog connects the MongoDB server during startup, it completes the following steps.
The AxoSyslog application connects the first address listed in the servers() option.
If the server is accessible and it is a master MongoDB server, AxoSyslog authenticates on the server (if needed), then starts sending the log messages to the server.
If the server is not accessible, or it is not a master server in a MongoDB replicaset and it does not send the address of the master server, AxoSyslog connects the next address listed in the servers() option.
If the server is not a master server in a MongoDB replicaset, but it sends the address of the master server, AxoSyslog connects the received address.
When AxoSyslog connects the master MongoDB server, it retrieves the list of replicas (from the replSet option of the server), and appends this list to the servers() option.
Warning
This means that AxoSyslog can send log messages to addresses that are not listed in its configuration.
Make sure to include the address of your master server in your AxoSyslog configuration file, otherwise you risk losing log messages if all the addresses listed in the AxoSyslog configuration are offline.
Addresses retrieved from the MongoDB servers are not stored, and can be lost when AxoSyslog is restarted. The retrieved addresses are not lost if the server() option of the destination was not changed in the configuration file since the last restart.
The failover mechanism used in the mongodb() driver is different from the client-side failover used in other drivers.
The AxoSyslog application attempts to connect another server if the servers() list contains at least two addresses, and one of the following events happens:
The safe-mode() option is set to no, and the MongoDB server becomes unreachable.
The safe-mode() option is set to yes, and AxoSyslog cannot insert a log message into the database because of an error.
In this case, AxoSyslog starts to connect the addresses in from the servers() list (starting from the first address) to find the new master server, authenticates on the new server (if needed), then continues to send the log messages to the new master server.
During this failover step, one message can be lost if the safe-mode() option is disabled.
If the original master becomes accessible again, AxoSyslog will automatically connect to the original master.
8.20.2 - mongodb() destination options
The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database.
The mongodb() destination has the following options:
bulk()
Type:
yes, no
Default:
yes
Available in AxoSyslog version 4.3.0 and newer.
Description: Enables bulk insert mode. If disabled, each messages is inserted individually.
Note: Bulk sending is only efficient if you use a constant collection (without templates), or the used template does not lead to too many collections switching within a reasonable time range.
Description: The name of the MongoDB collection where the log messages are stored (collections are similar to SQL tables). You can use templates to change the collection dynamically based on the source or the content of the message, for example, collection("${HOST}").
Warning
Hazard of data loss! The AxoSyslog application does not verify that the specified collection name does not contain invalid characters. If you specify a collection with an invalid name, the log messages sent to the MongoDB database will be irrevocably lost without any warning.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Sets the timezone used when expanding filename and tablename templates.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
on-error()
Type:
One of: drop-message, drop-property, fallback-to-string, silently-drop-message, silently-drop-property, silently-fallback-to-string
Default:
Use the global setting (which defaults to drop-message)
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
For MongoDB operations, AxoSyslog uses a one-minute timeout: if an operation times out, AxoSyslog assumes the operation has failed.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
Description: The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Structuring macros, metadata, and other value-pairs.
Note
Empty keys are not logged.
Note
Prior to version 4.0, AxoSyslog handled every data as strings, and allowed you to convert the strings into other types of data that only certain destinations data formats supported.
Starting with AxoSyslog 4.0, each name-value pair is a (name, type, value) triplet, and several components of AxoSyslog have typing support. For details, see Components that support data types.
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
The rest of this chapter and its sections build on your familiarity with the MQTT protocol, the concept of client and broker entities, and how these entities function within an MQTT system.
Example: Using the mqtt() destination in your configuration
The following example illustrates a mqtt() destination configured to fetch messages from the localhost:4444 address, and send them to the broker running on localhost:4445, using the mqtt test/test topic.
8.21.1 - Prerequisites to using the mqtt() destination
Using the current implementation of the mqtt() destination has the following prerequisites:
Installing the eclipse-paho-mqtt-c library.
Note
The default package manager for some Linux operating systems contains the eclipse-paho-mqtt-c library, but depending on your OS, you may have to install the library manually. For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.
Having a broker entity in a functional MQTT system.
Note
In your configuration, you will specify the broker entity of your MQTT system in the address() option of your mqtt() source.
8.21.2 - Limitations to using the mqtt() destination
Using the mqtt() destination of AxoSyslog has the following limitations:
You can only use the mqtt() destination with AxoSyslog version 3.33 or higher.
You cannot use the mqtt() destination without installing the the eclipse-paho-mqtt-c library.
For more information about how you can download and install the eclipse-paho-mqtt-c library, see Eclipse Paho on the Eclipse Foundation website.
The current implementation of the mqtt() destination supports versions 3.1 and 3.1.1 of the MQTT protocol.
8.21.3 - Options of the mqtt() destination
The mqtt() destination has the following options.
Required options: address(), fallback-topic(), and topic().
address()
Type:
string
Default:
tcp://localhost:1883
Required:
yes
Description: Specifies the hostname or IP address, and the port number of the MQTT broker to which AxoSyslog will send the log messages.
Syntax: <protocol type>://<host>:<port>
Supported protocol types: TCP, WS, SSL andWSS.
client-id()
Type:
string
Default:
syslog-ng-source-{topic option}
Required:
no
Description: The client-id() is used to identify the client to the MQTT server, which stores session data for each client. The session data can contains information regarding which message has been sent, received. It is not possible to set the client-id() to an empty string. To always start a new session see the cleansession() option.
cleansession()
Type:
`yes
Default:
no
Description: This option instruments the MQTT broker to clean the session data when connecting. The session data contains information about which message was processed.
fallback-topic()
Type:
string
Default:
N/A
Description: Required option when using templates in the topic() option.
If the resolved topic() template is not a valid topic, AxoSyslog will use the fallback-topic() option to send messages.
Note
If instead of strings, you use actual templates (that is, a macro like ${MESSAGE}, or a template function like $(format-json)) in the topic() option, configuring the fallback-topic() option is required.
Note
Occasionally, the reason why AxoSyslog cannot post messages to the configured topic() is that the topic contains invalid characters that originate from templates.
http-proxy()
Type:
URL
Default:
N/A
Description: Specifies HTTP/HTTPS proxy for WebSocket connections.
keep-alive()
Type:
positive integer number (in seconds)
Default:
60
Description: Specifies the number of seconds that AxoSyslog keeps the connection between the broker and clients open in case there is no message traffic. When keep-alive() number of seconds pass, the connection is terminated, and you have to reconnect.
On the MQTT side, the keep alive function provides a workaround method to access connections that are still open.
password()
Type:
string
Default:
N/A
Description: The password used to authenticate on the MQTT broker.
qos()
Type:
number
Default:
`0`
Possible values:
`0` - at most once (the fastest option)
`1` - at least once (a much slower option than `0`)
`2` - exactly once (the slowest option)
Description: The Quality of Service (QoS) level in MQTT messaging is an agreement between sender and receiver on the guarantee of delivering a message.
template()
Type:
string
Default:
$ISODATE $HOST $MSGHDR$MSG
Description: Specifies the message template that AxoSyslog sends to the MQTT broker.
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
The following options are relevant for the mqtt()tls() block: ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), ssl-version(), use-system-cert-store().
topic()
Type:
string or template
Default:
N/A
Description: Required option. Specifies the MQTT topic.
Note
The current implementation of the mqtt() destination does not support using the following characters for topic names:
$
+
#
username()
Type:
string
Default:
N/A
Description: The username used to authenticate on the MQTT broker.
8.21.4 - Possible error messages you may encounter while using the mqtt() destination
While using the mqtt() destination, you may encounter issues and corresponding error messages originating from the MQTT system. The following table contains the error messages you may encounter, the possible reasons behind them, and potential workaround methods.
Complete error message
Possible reason(s)
Possible solution(s)
ERROR, while init threaded dest.
The AxoSyslog application will not start.
You can try the following methods:
Restart AxoSyslog.
Stop some of the programs running on your computer.
Restart your computer, and then restart AxoSyslog.
mqtt: the topic() argument is required for mqtt destinations.
The topic() option is not set in your configuration. The AxoSyslog application will not start.
Set the missing topic() option in your configuration, then restart .
The mqtt destination does not support the batching of messages, ...
Your configuration may contain the batch-timeout() and / or batch-lines() options, which are not supported by the mqtt() destination. The AxoSyslog application will not start.
If your configuration contains the batch-timeout() and / or batch-lines() options, remove them from your configuration, and restart .
Disconnected during publish!
The AxoSyslog application can not send the message, because AxoSyslog disconnected from the broker. By default, AxoSyslog attempts to reconnect to the broker and send the messages 3 times.
If AxoSyslog fails all 3 attempts to reconnect to the broker and send the messages, you can try checking your configuration or restarting your MQTT system with AxoSyslog as a client.
Max message inflight! (publish)
The AxoSyslog application can not send the message due to the max message inflight broker response code (which signals that the broker has received too many messages, and it needs more time to process them). The AxoSyslog application will attempt to resend the message.
Wait until the broker can process the in-flight messages and AxoSyslog can attempt to resend the message.
Failure during publishing!
The AxoSyslog application can not send the message due to the failure broker response code. The AxoSyslog application will attempt to resend the message.
N/A
Error during publish!
The AxoSyslog application can not send the message, and drops it.
Possible reason: bad_utf8_string (topic), NULL parameter`.
That is, the most probable reasons behind this issue are either that the topic name in your configuration is not correct, or that the message field is empty.
You can try the following methods:
Modify the name of the topic() option in your configuration.
Make sure that the message field is not empty.
Disconnected while waiting the response!
The AxoSyslog application has sent the message, but the client disconnected from the broker before AxoSyslog received the response. The AxoSyslog application will attempt to reconnect, or to resend the message.
The AxoSyslog application will attempt to reconnect to the broker and send the in-flight message. If the reconnect attempt fails, AxoSyslog will resend the message.
"Error while waiting the response!"
The AxoSyslog application can not get any response from the broker, due to the failure broker response code. The AxoSyslog will attempt to resend the message.
In this case, you will receive a further error message, depending on what the problem is. Wait for the second error message for more information about how you can proceed.
"Error constructing topic ..."
Due to an issue with the configured topic template, the mqtt() destination will use the fallback-topic() option instead.
N/A
mqtt dest: topic name is illegal, it can't be empty
This error message is related to the "Error constructing topic ..." error message.
In this case, the topic template returns a 0 length string. As a result, the mqtt() destination will use the fallback-topic() option instead.
N/A
Error connecting mqtt client ...
The AxoSyslog application can not connect to broker, and it will attempt to reconnect later.
If the issue persists, you can try the following:
Update your eclipse-paho-mqtt-c library.
Restart AxoSyslog.
Error creat mqtt client ...
The AxoSyslog application encountered an error while creating the MQTT client, and it will attempt to create it later.
Possible reasons:
There is a wrong address() set in your configuration.
The broker is not running.
You can try the following methods:
Check the address() option in your configuration, and modify if necessary.
Check if the specified broker is running by connecting to it manually, and then sending the broker a message.
8.22 - network: Send messages to a remote log server using the RFC3164 protocol (network() driver)
The network() destination driver can send syslog messages conforming to RFC3164 from the network using the TCP, TLS, and UDP networking protocols.
UDP is a simple datagram oriented protocol, which provides “best effort service” to transfer messages between hosts. It may lose messages, and no attempt is made to retransmit lost messages. The BSD-syslog protocol traditionally uses UDP.
Use UDP only if you have no other choice.
TCP provides connection-oriented service: the client and the server establish a connection, each message is acknowledged, and lost packets are resent. TCP can detect lost connections, and messages are lost, only if the TCP connection breaks. When a TCP connection is broken, messages that the client has sent but were not yet received on the server are lost.
The AxoSyslog application supports TLS (Transport Layer Security, also known as SSL) over TCP. For details, see Encrypting log messages with TLS.
Declaration:
network("<destination-address>"[options]);
The network() destination has a single required parameter that specifies the destination host address where messages should be sent. If name resolution is configured, you can use the hostname of the target server. By default, AxoSyslog sends messages using the TCP protocol to port 514.
Example: Using the network() driver
TCP destination that sends messages to 10.1.2.3, port 1999:
To send messages using the IETF-syslog message format without using the IETF-syslog protocol, enable the syslog-protocol flag. (For details on how to use the IETF-syslog protocol, see syslog() destination options.)
The network() driver sends messages to a remote host (for example, a server or relay) on the local intranet or internet using the RFC3164 syslog protocol (for details about the protocol, see BSD-syslog or legacy-syslog messages). The network() driver supports sending messages using the UDP, TCP or the encrypted TLS networking protocols.
These destinations have the following options:
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
close-on-input()
Type:
yes
Default:
yes
Description: By default, AxoSyslog closes destination sockets if it receives any input from the socket (for example, a reply). If this option is set to no, AxoSyslog just ignores the input, but does not close the socket.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Available only in AxoSyslog version 3.17 and later. For details about how client-side failover works, see Client-side failover.
servers()
Type:
list of IP addresses and fully-qualified domain names
Default:
empty
Description: Specifies a secondary destination server where log messages are sent if the primary server becomes inaccessible. To list several failover servers, separate the address of the servers with comma. By default, AxoSyslog waits for the a server before switching to the next failover server is set in the time-reopen() option.
If failback() is not set, AxoSyslog does not attempt to return to the primary server even if it becomes available. In case the failover server fails, AxoSyslog attempts to connect the next failover server in the list in round-robin fashion.
Warning
The failover servers must be accessible on the same port as the primary server.
failback()
Description: Available only in AxoSyslog version 3.17 and later.
When AxoSyslog starts up, it always connects to the primary server first. In the failover() option there is a possibility to customize the failover modes.
Depending on how you set the failback() option, AxoSyslog behaves as follows:
round-robin mode: If failback() is not set, AxoSyslog does not attempt to return to the primary server even if it becomes available. In case the failover server fails, AxoSyslog attempts to connect the next failover server in the list in round-robin fashion.
In the following example AxoSyslog handles the logservers in round-robin fashion if the primary logserver becomes inaccessible (therefore failback() option is not set).
failback mode: If failback() is set, AxoSyslog attempts to return to the primary server.
After AxoSyslog connects a secondary server during a failover, it sends a probe every tcp-probe-interval() seconds towards the primary server. If the primary logserver responds with a TCP ACK packet, the probe is successful. When the number of successful probes reaches the value set in the successful-probes-required() option, AxoSyslog tries to connect the primary server using the last probe.
Note
AxoSyslog always waits for the result of the last probe before sending the next message. So if one connection attempt takes longer than the configured interval, that is, it waits for connection time out, you may experience longer intervals between actual probes.
In the following example AxoSyslog attempts to return to the primary logserver, as set in the failback() option: it will check if the server is accessible every tcp-probe-interval() seconds, and reconnect to the primary logserver after three successful connection attempts.
Default value for tcp-probe-interval(): 60 seconds
Default value for successful-probes-required(): 3
Example: Configuring failover servers
In the following example AxoSyslog handles the logservers in round-robin fashion if the primary logserver becomes uneccassible (therefore failback() option is not set).
In the following example AxoSyslog attempts to return to the primary logserver, as set in the failback() option: it will check if the server is accessible every tcp-probe-interval() seconds, and reconnect to the primary logserver after three successful connection attempts.
This option is not available for the connection-less UDP protocol, because in this case the client does not detect that the destination becomes inaccessible.
flags()
Type:
no-multi-line, syslog-protocol
Default:
empty set
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
flush-lines()
Type:
number
Default:
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).
Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).
For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.
Description: Specifies the Type-of-Service value of outgoing packets.
ip-ttl()
Type:
number
Default:
0
Description: Specifies the Time-To-Live value of outgoing packets.
keep-alive()
Type:
yes or no
Default:
yes
Description: Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.
localip()
Type:
string
Default:
0.0.0.0
Description: The IP address to bind to before connecting to target.
localport()
Type:
number
Default:
0
Description: The port number to bind to. Messages are sent from this port.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
port() or destport()
Type:
number
Default:
601
Description: The port number to connect to. Note that the default port numbers used by AxoSyslog do not comply with the latest RFC which was published after the release of AxoSyslog 3.0.2, therefore the default port numbers will change in the future releases.
so-broadcast()
Type:
yes or no
Default:
no
Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.
so-keepalive()
Type:
yes or no
Default:
no
Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-rcvbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.
so-sndbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.
spoof-source()
Type:
yes or no
Default:
no
Description: Enables source address spoofing. This means that the host running AxoSyslog generates UDP packets with the source IP address matching the original sender of the message. It is useful when you want to perform some kind of preprocessing using AxoSyslog then forward messages to your central log management solution with the source address of the original sender. This option only works for UDP destinations though the original message can be received by TCP as well. This option is only available if syslog-ng was compiled using the --enable-spoof-source configuration option.
The maximum size of spoofed datagrams in udp() destinations is set to 1024 bytes by default. To change the maximum size, use the spoof-source-max-msglen() option.
Note
Anything above the size of the maximum transmission unit (MTU), which is 1500 bytes by default, is not recommended because of fragmentation.
The maximum datagram in IP protocols (both IPv4 and IPv6) is 65535 bytes including the IP and UDP headers. The minimum size of the IPv4 header is 20 bytes, the IPv6 is 40 bytes, and the UDP is 8 bytes.
Warning
To use spoofing on Microsoft Windows platforms, you must also set the spoof-interface() option as well.
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
tcp-keepalive-intvl()
Type:
number [seconds]
Default:
0
Description: Specifies the interval (number of seconds) between subsequential keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.
Warning
The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
tcp-keepalive-probes()
Type:
number
Default:
0
Description: Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.
Warning
The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
tcp-keepalive-time()
Type:
number [seconds]
Default:
0
Description: Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.
Warning
<p>The <code>tcp-keepalive-time()</code>, <code>tcp-keepalive-probes()</code>, and <code>tcp-keepalive-intvl()</code> options only work on platforms which support the <code>TCP_KEEPCNT</code>, <code>TCP_KEEPIDLE</code>,and <code>TCP_KEEPINTVL</code> setsockopts. Currently, this is Linux.</p>
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
Note
If a message uses the IETF-syslog format (RFC5424), only the text of the message can be customized (that is, the $MESSAGE part of the log), the structure of the header is fixed.
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
transport()
Type:
udp, tcp, or tls
Default:
tcp
Description: Specifies the protocol used to send messages to the destination server.
If you use the udp transport, AxoSyslog automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
8.23 - Send messages to OpenObserve
Starting with version 4.5.0, AxoSyslog can send messages to OpenObserve using its Logs Ingestion - JSON API. This API accepts multiple records in batch in JSON format.
To configure AxoSyslog, you’ll need the username, password, the name of your organization, and the name of the OpenObserve stream where you want to send your data.
This driver is actually a reusable configuration snippet configured to send log messages using the http() driver using a template. You can find the source of this configuration snippet on GitHub.
Options
The following options are specific to the openobserve-log() destination. But since this destination is based on the http() destination, you can use the options of the http() destination as well if needed.
Note: The openobserve-log() destination automatically configures some of these http() destination options as required by the OpenObserve Ingest API.
Description: The password for the username specified in the user() option.
port()
Type:
integer
Default:
5080
Description: The port number of the server.
record()
Type:
string
Default:
"--scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}"
Description: A JSON object representing key-value pairs sent to OpenObserve, formatted as AxoSyslog value-pairs. By default, the openobserve-log() destination sends the RFC5424 fields as attributes. If you want to send different fields, override the default content of the record() field.
stream()
Type:
string
Default:
"default"
Description: The OpenObserve stream where AxoSyslog sends the data, for example, your-example-stream.
user()
Type:
string
Default:
-
Description: The username of the account, for example, [email protected].
url()
Type:
string
Default:
-
Description: The base URL of the OpenObserve Ingest API. The actual URL is constructed from the base URL and some other options of the destination: url():port()/api/organization()/stream()/_json
8.24 - opensearch: Send messages to OpenSearch
Available in AxoSyslog version 4.4 and later.
The opensearch() destination can directly post log messages to OpenSearch using its HTTP endpoint.
HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format.
This driver is actually a reusable configuration snippet configured to send log messages using the http() driver using a template. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
8.24.1 - Batch mode and load balancing
The opensearch() destination automatically sends multiple log messages in a single HTTP request, increasing the rate of messages that your Elasticsearch deployment can consume. For details on adjusting and fine-tuning the batch mode, see the following section.
Batch size
The batch-bytes(), batch-lines(), and batch-timeout() options of the destination determine how many log messages AxoSyslog sends in a batch. The batch-lines() option determines the maximum number of messages AxoSyslog puts in a batch in. This can be limited based on size and time:
AxoSyslog sends a batch every batch-timeout() milliseconds, even if the number of messages in the batch is less than batch-lines(). That way the destination receives every message in a timely manner even if suddenly there are no more messages.
AxoSyslog sends the batch if the total size of the messages in the batch reaches batch-bytes() bytes.
To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-bytes(), batch-lines(), batch-timeout() options.
Example: HTTP batch mode
In the following example, a batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).
Starting with version 3.19, you can specify multiple URLs, for example, url("site1" "site2"). In this case, AxoSyslog sends log messages to the specified URLs in a load-balance fashion. This means that AxoSyslog sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order AxoSyslog has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.
Warning
If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.
Starting with version AxoSyslog version 3.22, you can use any of the following formats to specify multiple URLs:
url("server1", "server2", "server3");# comma-separated strings url("server1""server2""server3");# space-separated strings url("server1 server2 server3");# space-separated within a single string
Example: HTTP load balancing
The following destination sends log messages to 3 different indexer nodes. Each node is assigned a separate worker thread. A batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).
If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1", "site2", "site3")), set the workers() option to 3 or more.
8.24.2 - opensearch() destination options
The opensearch destination of AxoSyslog can directly post log messages to an OpenSearch deployment using the OpenSearch Bulk API over the HTTP and Secure HTTP (HTTPS) protocols. The opensearch destination has the following options. The required options are: index() and url().
This destination is available in AxoSyslog version 4.4 and later.
batch-bytes()
Accepted values:
number [bytes]
Default:
none
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Name of a file that contains an X.509 CA certificate (or a certificate chain) in PEM format. The AxoSyslog application uses this certificate to validate the certificate of the HTTPS server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Name of a file, that contains an X.509 certificate (or a certificate chain) in PEM format, suitable as a TLS certificate, matching the private key set in the key-file() option. The AxoSyslog application uses this certificate to authenticate the AxoSyslog client on the destination server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Depends on the OpenSSL version that AxoSyslog uses
Description: Specifies the cipher, hash, and key-exchange algorithms used for the encryption, for example, ECDHE-ECDSA-AES256-SHA384. The list of available algorithms depends on the version of OpenSSL used to compile AxoSyslog. To specify multiple ciphers, separate the cipher names with a colon, and enclose the list between double-quotes, for example:
For a list of available algorithms, execute the openssl ciphers -v command. The first column of the output contains the name of the algorithms to use in the cipher-suite() option, the second column specifies which encryption protocol uses the algorithm (for example, TLSv1.2). That way, the cipher-suite() also determines the encryption protocol used in the connection: to disable SSLv3, use an algorithm that is available only in TLSv1.2, and that both the client and the server supports. You can also specify the encryption protocols using ssl-options().
You can also use the following command to automatically list only ciphers permitted in a specific encryption protocol, for example, TLSv1.2:
echo"cipher-suite(\"$(openssl ciphers -v | grep TLSv1.2 | awk '{print $1}'| xargs echo -n | sed 's/ /:/g'| sed -e 's/:$//')\")"
Note that starting with version 3.10, when AxoSyslog receives TLS-encrypted connections, the order of ciphers set on the AxoSyslog server takes precedence over the client settings.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Sets the specified value as the ID of the OpenSearch index (_id).
delimiter()
Accepted values:
string
Default:
newline character
Description: By default, AxoSyslog separates the log messages of the batch with a newline character. You can specify a different delimiter by using the delimiter() option.
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The name of the OpenSearch index where OpenSearch will store the messages received from AxoSyslog. This option is mandatory for this destination.
You can use macros and template functions, but you must ensure that the resolved template contains only characters that OpenSearch permits in the name of the index. The AxoSyslog application does not validate the name of the index. For details on the characters permitted in the name of OpenSearch indices, see the documentation of OpenSearch.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
key-file()
Accepted values:
Filename
Default:
none
Description: The name of a file that contains an unencrypted private key in PEM format, suitable as a TLS key. If properly configured, the AxoSyslog application uses this private key and the matching certificate (set in the cert-file() option) to authenticate the AxoSyslog client on the destination server.
This destination supports only unencrypted key files (that is, the private key cannot be password-protected).
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: The password that AxoSyslog uses to authenticate on the server where it sends the messages.
peer-verify()
Accepted values:
`yes
Default:
yes
Description: Verification method of the peer. The following table summarizes the possible options and their results depending on the certificate of the peer.
The remote peer has:
no certificate
invalid certificate
valid certificate
Local peer-verify() setting
no (optional-untrusted)
TLS-encryption
TLS-encryption
TLS-encryption
yes (required-trusted)
rejected connection
rejected connection
TLS-encryption
For untrusted certificates only the existence of the certificate is checked, but it does not have to be valid — AxoSyslog accepts the certificate even if it is expired, signed by an unknown CA, or its CN and the name of the machine mismatches.
Warning
When validating a certificate, the entire certificate chain must be valid, including the CA certificate. If any certificate of the chain is invalid, AxoSyslog will reject the connection.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
proxy()
Type:
The proxy server address, in `proxy("PROXY_IP:PORT")` format.
For example, `proxy("http://myproxy:3128")`
Default:
None
Description:
You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.
Alternatively, you can leave the HTTP as-is, in which case the driver leaves the default http_proxy and https_proxy environment variables unmodified.
Note
Configuring the proxy() option overwrites the default http_proxy and https_proxy environment variables.
Example: the proxy() option in configuration
The following example illustrates including the proxy() option in your configuration.
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
To handle HTTP error responses, if the HTTP server returns 5xx codes, AxoSyslog will attempt to resend messages until the number of attempts reaches retries. If the HTTP server returns 4xx codes, AxoSyslog will drop the messages.
ssl-version()
Type:
string
Default:
None, uses the libcurl default
Description: Specifies the permitted SSL/TLS version. Possible values: sslv2, sslv3, tlsv1, tlsv1_0, tlsv1_1, tlsv1_2, tlsv1_3.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
timeout()
Type:
number [seconds]
Default:
10
Description: The value (in seconds) to wait for an operation to complete, and attempt to reconnect the server if exceeded.
url()
Type:
URL or list of URLs, for example, url(“site1” “site2”)
Default:
N/A
Description: Specifies the hostname or IP address and optionally the port number of the OpenSearch indexer. Use a colon (:) after the address to specify the port number of the server. For example: http://your-opensearch-indexer.server:8088/_bulk
This option is mandatory for this destination.
Make sure that the URL ends with _bulk, this is the OpenSearch API endpoint that properly parses the messages sent by AxoSyslog.
In case the server on the specified URL returns a redirect request, AxoSyslog automatically follows maximum 3 redirects. Only HTTP and HTTPS based redirections are supported.
Starting with version 3.19, you can specify multiple URLs, for example, url("site1" "site2"). In this case, AxoSyslog sends log messages to the specified URLs in a load-balance fashion. This means that AxoSyslog sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order AxoSyslog has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.
Warning
If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.
Starting with version AxoSyslog version 3.22, you can use any of the following formats to specify multiple URLs:
url("server1", "server2", "server3");# comma-separated strings url("server1""server2""server3");# space-separated strings url("server1 server2 server3");# space-separated within a single string
user()
Type:
string
Default:
Description: The username that AxoSyslog uses to authenticate on the server where it sends the messages.
use-system-cert-store()
Type:
`yes
Default:
no
Description: Use the certificate store of the system for verifying HTTPS certificates. For details, see the curl documentation.
workers()
Type:
integer
Default:
4
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1", "site2", "site3")), set the workers() option to 3 or more.
8.25 - osquery: Send log messages to osquery's syslog table
The osquery() driver sends log messages to osquery’s syslog table.
The syslog table contains logs forwarded over a named pipe from syslog-ng. When an osquery process that supports the syslog table starts up, it creates (and properly sets permissions for) a named pipe for AxoSyslog to write to.
The osquery() destination has the following options:
file()
Type:
string
Default:
N/A
Description: Specifies a path to the file where log messages are stored, for example, for debug purposes.
Specifying this option is optional. However, when you start losing logs for some reason, then it is recommended to write outgoing log messages to a specified file, in the same format that messages are written to the pipe. You can also use a template() function called t_osquery, which re-formats messages so they comply with the text-based protocol that osquery accepts.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Specifies a custom path to the named pipe that acts as the interface between osquery and syslog-ng. (The default path is set in the SCL file.)
Specifying this option is optional.
8.26 - Send logs, metrics, and traces to OpenTelemetry
Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.
tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.
batch-bytes()
Accepted values:
number [bytes]
Default:
4MB
Available in AxoSyslog version 4.6 and later.
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option. The batch might be at most 1 message larger than the set limit.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
OTLP has a default 4 MiB batch limit, therefore the default value for batch-bytes() is 4 MB, which is a bit below 4 MiB.
The batch size is calculated before compression, which is the same as the limit is calculated on the server.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
channel-args()
Type:
See description
Default:
-
Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:
Description: Enables compression in gRPC requests. Although gRPC supports various compression methods, currently only deflate is supported (which is basically the same as gzip).
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
8.27 - pipe: Send messages to named pipes
The pipe() driver sends messages to a named pipe like /dev/xconsole.
The pipe driver has a single required parameter, specifying the filename of the pipe to open. The filename can include macros. For the list of available optional parameters, see pipe() destination options.
Declaration:
pipe(filename);
Warning
Starting with AxoSyslog 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.
Example: Using the pipe() driver
destination d_pipe { pipe("/dev/xconsole");};
8.27.1 - pipe() destination options
This driver sends messages to a named pipe like /dev/xconsole.
The pipe() destination has the following options:
create-dirs()
Type:
yes or no
Default:
no
Description: Enable creating non-existing directories when creating files or socket files.
flags()
Type:
no-multi-line, syslog-protocol
Default:
empty set
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
flush-lines()
Type:
number
Default:
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
group()
Type:
string
Default:
Use the global settings
Description: Set the group of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: group().
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The number of messages that the output queue can store.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
owner()
Type:
string
Default:
Use the global settings
Description: Set the owner of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: owner().
pad-size()
Type:
number
Default:
0
Description: If set, AxoSyslog will pad output messages to the specified size (in bytes). Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes).
Warning
Hazard of data loss! If the size of the incoming message is larger than the previously set pad-size() value, AxoSyslog will truncate the message to the specified size. Therefore, all message content above that size will be lost.
perm()
Type:
number (octal notation)
Default:
0600
Description: The permission mask of the pipe. For octal numbers prefix the number with ‘0’, for example: use 0755 for rwxr-xr-x.
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reap()
Accepted values:
number (seconds)
Default:
60 or 0, see description for details
Description: The time to wait in seconds before an idle destination file or pipe is closed. Note that only destination files having macros in their filenames are closed automatically.
Starting with version 3.23, the way how time-reap() works is the following.
If the time-reap() option of the destination is set, that value is used, for example:
destination d_fifo { pipe("/tmp/test.fifo",
time-reap(30)# sets time-reap() for this destination only);};
If the time-reap() option of the destination is not set, and the destination does not use a template or macro in its filename or path, time-reap() is automatically set to 0. For example:
destination d_fifo { pipe("/tmp/test.fifo",
);};
Otherwise, the value of the global time-reap() option is used, which defaults to 60 seconds.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
8.28 - program: Send messages to external applications
The program() driver starts an external application or script and sends the log messages to its standard input (stdin). Usually, every message is a single line (ending with a newline character), which your script can process. Make sure that your script runs in a loop and keeps reading the standard input — it should not exit. (If your script exits, AxoSyslog tries to restart it.)
The program() driver has a single required parameter, specifying a program name to start. The program is executed with the help of the current shell, so the command may include both file patterns and I/O redirections. For the list of available optional parameters, see program() destination options.
Declaration:
program(command_to_run);
When using the program() driver, consider the following:
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor, you might have to modify your AppArmor configuration to enable AxoSyslog to execute external applications.
The AxoSyslog application executes program destinations through the standard system shell. If the system shell is not bash and you experience problems with the program destination, try changing the /bin/sh link to /bin/bash.
If the external program exits, the AxoSyslog application automatically restarts it. However it is not recommended to launch programs for single messages, because if the message rate is high, launching several instances of an application might overload the system, resulting in Denial of Service.
When the AxoSyslog application stops, it will automatically stop the external program. To avoid restarting the application when AxoSyslog is only reloaded, enable the keep-alive() option in the program destination.
Certain external applications buffer the log messages, which might cause unexpected latency and other problems. For example, if you send the log messages to an external Perl script, Perl uses a line buffer for terminal output and block buffer otherwise. You might want to disable buffering in the external application.
Example: Using the program() destination driver
The message format does not include the priority and facility values by default. To add these values, specify a template for the program destination, as shown in the following example. Make sure to end your template with a newline character (\n).
The following shell script writes the incoming messages into the /tmp/testlog file.
#!/bin/bashwhileread line ;doecho$line >> /tmp/testlog
done
8.28.1 - program() destination options
This driver starts an external application or script and sends the log messages to its standard input (stdin).
The program() destination has the following options:
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
flush-lines()
Type:
number
Default:
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The number of messages that the output queue can store.
inherit-environment()
Type:
yes
Default:
yes
Description: By default, when program() starts an external application or script, it inherits the entire environment of the parent process (that is, AxoSyslog). Use inherit-environment(no) to prevent this.
keep-alive()
Type:
yes or no
Default:
no
Description: Specifies whether the external program should be closed when AxoSyslog is reloaded.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
Note that in earlier versions of AxoSyslog, the default for the mark-mode of the program destination was none. Now it defaults to the global setting, so the program destination will emit a MARK message every mark-freq interval. To avoid such messages, set the mark-mode() option of the destination to none.
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
Make sure to end your template with a newline character (\n).
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
8.29 - pseudofile()
The pseudofile() destination driver is a very simple driver, aimed at delivering messages to special files such as files in the /proc, /dev or /sys directories. It opens and closes the file after each write operation, instead of keeping it open. It does not append further data. It does not support templates in the filename, and does not have a queue, processing is performed immediately as read by the source. Therefore, no loss is possible, but it takes CPU time from the source, so it is not adequate in high-traffic situations.
Declaration:
pseudofile(filename options());
8.29.1 - pseudofile() destination options
The pseudofile() destination has the following options:
file()
Type:
filename with path
Default:
Description: The file to write messages to, including the path.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
8.30 - python: Write custom Python destinations
The Python destination allows you to write your own destination in Python. You can import external Python modules to process the messages, and send them to other services or servers. Since many services have a Python library, the Python destination makes integrating AxoSyslog very easy and quick.
The following points apply to using Python blocks in AxoSyslog in general:
Python parsers and template functions are available in AxoSyslog version 3.10 and later.
Python destinations and sources are available in AxoSyslog version 3.18 and later.
Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).
The Python block must be a top-level block in the AxoSyslog configuration file.
If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.
In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".
The Python object is initiated every time when AxoSyslog is started or reloaded.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
The Python block can contain multiple Python functions.
Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.
Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.
Python error messages are available in the internal() source of AxoSyslog.
You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.
Note
Starting with 3.26, AxoSyslog assigns a persist name to Python sources and destinations. The persist name is generated from the class name. If you want to use the same Python class multiple times in your AxoSyslog configuration, add a unique persist-name() to each source or destination, otherwise AxoSyslog will not start. For example:
Alternatively, you can include the following line in the Python package: @staticmethod generate_persist_name. For example:
from syslogng import LogSource
class PyNetworSource(LogSource):
@staticmethod
def generate_persist_name(options):
return options["port"] def run(self):
pass
def request_exit(self):
pass
Declaration:
Python destinations consist of two parts. The first is a AxoSyslog destination object that you define in your AxoSyslog configuration and use in the log path. This object references a Python class, which is the second part of the Python destination. The Python class processes the log messages it receives, and can do virtually anything that you can code in Python. You can either embed the Python class into your AxoSyslog configuration file, or store it in an external Python file.
destination <name_of_the_python_destination>{ python( class("<name_of_the_python_class_executed_by_the_destination>"));}; python { class <name_of_the_python_class_executed_by_the_destination>(object):
def open(self):
"""Open a connection to the target service
Should return False if opening fails"""return True
def close(self):
"""Close the connection to the target service""" pass
def is_opened(self):
"""Check if the connection to the target is able to receive messages"""return True
def init(self, options):
"""This method is called at initialization time
Should return false if initialization fails"""return True
def deinit(self):
"""This method is called at deinitialization time""" pass
def send(self, msg):
"""Send a message to the target service
It should return True to indicate success. False will suspend the
destination for a period specified by the time-reopen() option."""return True
def flush(self):
"""Flush the queued messages""" pass
};
The AxoSyslog application initializes Python objects every time when it is started or reloaded. The init method is executed as part of the initialization. You can perform any initialization steps that are necessary for your source to work.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
When this method returns with False, AxoSyslog does not start. It can be used to check options and return False when they prevent the successful start of the source.
options: This optional argument contains the contents of the options() parameter of the AxoSyslog configuration object as a Python dictionary.
The open(self) method opens the resources required for the destination, for example, it initiates a connection to the target service. It is called after init() when AxoSyslog is started or reloaded. If send() returns with an error, AxoSyslog calls close() and open() before trying to send again.
If open() fails, it should return the False value. In this case, AxoSyslog retries it every time-reopen() seconds. By default, this is 1 second for Python sources and destinations, the value of time-reopen() is not inherited from the global option. For details, see Error handling in the python() destination.
send(self, message) method (mandatory)
The send method sends a message to the target service. It should return True to indicate success, or self.QUEUED when using batch mode. For other possible return values, see the description of the flush() method. Note that for batch mode, the flush() method must be implemented as well.
This is the only mandatory method of the destination.
If a message cannot be delivered after the number of times set in retries() (by default: 3), AxoSyslog drops the message and continues with the next message. For details, see Error handling in the python() destination.
The method can return True, False, or one of the following constants:
self.DROP: The message is dropped immediately.
self.ERROR: Corresponds to boolean False. The message is put back to the queue, and sending the message is attempted (up to the number of the retries() option). The destination is suspended for time-reopen() seconds.
self.SUCCESS: Corresponds to boolean True. The message was sent successfully.
self.QUEUED: The send() method should return this value when using batch mode, if it has successfully added the message to the batch. Message acknowledgment of batches is controlled by the flush() method.
self.NOT_CONNECTED: The message is put back to the queue, and the destination is suspended. The open() method will be called, and the sending the messages will be continued with the same message/batch.
self.RETRY: The message is put back to the queue, and sending the message is attempted (up to the number of the retries() option). If sending the message has failed retries() times, self.NOT_CONNECTED is returned.
flush(self) method (optional)
Send the messages in a batch. You can use this method to implement batch-mode message sending instead of sending messages one-by-one. When using batch mode, the send() method adds the messages to a batch (for example, a list), and the flush() method sends the messages as configured in the batch-bytes(), batch-lines(), or batch-timeout() options.
The method can return True, False, or one of the following constants:
self.DROP: The messages cannot be sent and the entire batch is dropped immediately.
self.ERROR: Corresponds to boolean False. The message is put back to the queue, and sending the message is attempted (up to the number of the retries() option). The destination is suspended for time-reopen() seconds.
self.SUCCESS: Corresponds to boolean True. The message was sent successfully.
self.NOT_CONNECTED: The message is put back to the queue, and the destination is suspended. The open() method will be called, and the sending the messages will be continued with the same message/batch.
self.RETRY: The message is put back to the queue, and sending the message is attempted (up to the number of the retries() option). If sending the message has failed retries() times, self.NOT_CONNECTED is returned.
close(self) method (optional)
Close the connection to the target service. Usually it is called right before deinit() when stopping or reloading AxoSyslog. It is also called when send() fails.
The deinit(self) method (optional)
This method is executed when AxoSyslog is stopped or reloaded. This method does not return a value.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
Error handling in the python() destination
The Python destination handles errors as follows.
Currently AxoSyslog ignores every error from the open method until the first log message arrives to the Python destination. If the fist message has arrived and there was an error in the open method, AxoSyslog starts calling the open method every time-reopen() second, until opening the destination succeeds.
If the open method returns without error, AxoSyslog calls the send method to send the first message.
If the send method returns with an error, AxoSyslog calls the is_opened method.
If the is_opened method returns an error, AxoSyslog starts calling the open method every time-reopen() second, until opening the destination succeeds.
Otherwise, AxoSyslog calls the send method again.
If the send method has returned with an error retries() times and the is_opened method has not returned any errors, AxoSyslog drops the message and attempts to process the next message.
Example: Write logs into a file
The purpose of this example is only to demonstrate the basics of the Python destination, if you really want to write log messages into text files, use the file destination instead.
The following sample code writes the body of log messages into the /tmp/example.txt file. Only the send() method is implemented, meaning that AxoSyslog opens and closes the file for every message.
The Python destination allows you to write your own destination in Python. The python() destination has the following options. The class() option is mandatory. For details on writing destinations in Python, see python: Write custom Python destinations.
batch-bytes()
Accepted values:
number [bytes]
Default:
none
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
This option does not have any effect unless the flush() method is implemented in the destination.
batch-lines()
Type:
number
Default:
25
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
This option does not have any effect unless the flush() method is implemented in the destination.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
This option does not have any effect unless the flush() method is implemented in the destination.
class()
Type:
string
Default:
N/A
Description: The name of the Python class that implements the destination, for example:
python( class("MyPythonDestination"));
If you want to store the Python code in an external Python file, the class() option must include the name of the Python file containing the class, without the path and the .py extension, for example:
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
loaders()
Type:
list of python modules
Default:
empty list
Description: The AxoSyslog application imports Python modules specified in this option, before importing the code of the Python class. This option has effect only when the Python class is provided in an external Python file. This option has no effect when the Python class is provided within the AxoSyslog configuration file (in a python{} block). You can use the loaders() option to modify the import mechanism that imports Python class. For example, that way you can use hy in your Python class.
python(class(usermodule.HyParser) loaders(hy))
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
on-error()
Type:
One of: drop-message, drop-property, fallback-to-string, silently-drop-message, silently-drop-property, silently-fallback-to-string
Default:
Use the global setting (which defaults to drop-message)
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
options()
Type:
string
Default:
N/A
Description: This option allows you to pass custom values from the configuration file to the Python code. Enclose both the option names and their values in double-quotes. The Python code will receive these values during initialization as the options dictionary. For example, you can use this to set the IP address of the server from the configuration file, so it is not hard-coded in the Python object.
For example, you can refer to the value of the host field in the Python code as options["host"]. Note that the Python code receives the values as strings, so you might have to cast them to the type required, for example: int(options["port"])
Note
From version 3.27, AxoSyslog supports the arrow syntax for declaring custom Java and Python options. You can alternatively declare them using a similar syntax:
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
Note
Starting with 3.26, AxoSyslog assigns a persist name to Python sources and destinations. The persist name is generated from the class name. If you want to use the same Python class multiple times in your AxoSyslog configuration, add a unique persist-name() to each source or destination, otherwise AxoSyslog will not start. For example:
Alternatively, you can include the following line in the Python package: @staticmethod generate_persist_name. For example:
from syslogng import LogSource
class PyNetworSource(LogSource):
@staticmethod
def generate_persist_name(options):
return options["port"] def run(self):
pass
def request_exit(self):
pass
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
1
Description: The time to wait in seconds before a dead connection is reestablished.
value-pairs()
Type:
parameter list of the value-pairs() option
Default:
scope("selected-macros" "nv-pairs")
Description: The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Structuring macros, metadata, and other value-pairs.
Note
Empty keys are not logged.
You can use this option to limit which name-value pairs are passed to the Python code for each message. Note that if you use the value-pairs() option, the Python code receives the specified value-pairs as a Python dict. Otherwise, it receives the message object. In the following example, only the text of the log message is passed to Python.
Starting with version 3.34, you can send multiple log messages with the help of Redis’s pipelining feature.
Batch size
The batch-lines(), batch-lines(), and batch-timeout() options of the destination determine how many log messages AxoSyslog sends in a batch. The batch-lines() option determines the maximum number of messages AxoSyslog puts in a batch in. This can be limited based on size and time:
AxoSyslog sends a batch every batch-timeout() milliseconds, even if the number of messages in the batch is less than batch-lines(). That way the destination receives every message in a timely manner even if suddenly there are no more messages.
To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-lines() and/or batch-timeout() options.
Example: Redis batch mode
The following destination sends log messages to a Redis server using the pipelining feature. A batch consists of 100 messages and is sent every 10 seconds (10000 milliseconds) if there is less than 100 messages are in the queue.
The redis() driver sends messages as name-value pairs to a Redis key-value store.
The redis() destination has the following options:
auth()
Type:
hostname or IP address
Default:
N/A
Description: The password used for authentication on a password-protected Redis server. Available in AxoSyslog version 3.10 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
command()
Type:
comma-separated list of strings ("", “«”, “<s<o<-command-parameter>”, “<th<d-”)
Default:
empty string
Description: The Redis command to execute, for example, LPUSH, INCR, or HINCRBY. Using the HINCRBY command with an increment value of 1 allows you to create various statistics. For example, the command("HINCRBY" "${HOST}/programs" "${PROGRAM}" "1") command counts the number of log messages on each host for each program.
Note the following points when using the redis() destination:
You can use macros and templates in the parameters of the Redis command.
Currently you can use only one command in a redis() destination.
The AxoSyslog application ignores the return value of the command. If the Redis server returns an error, AxoSyslog closes the connection.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The hostname or IP address of the Redis server.
port()
Type:
number
Default:
6379
Description: The port number of the Redis server.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
8.32 - riemann: Monitor your data with Riemann
The riemann() driver sends your data (for example, metrics or events) to a Riemann monitoring system.
The riemann() driver sends metrics or events to a Riemann monitoring system.
The riemann() destination has the following options:
attributes()
Type:
parameter list of the value-pairs() option
Default:
Description: The attributes() option adds extra metadata to the Riemann event, that can be displayed on the Riemann dashboard. To specify the metadata to add, use the syntax of the value-pairs() option. For details on using value-pairs(), see Structuring macros, metadata, and other value-pairs.
description()
Type:
template, macro, or string
Default:
Description: The value to add as the description field of the Riemann event.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Instead of the arrival time into Riemann, AxoSyslog can also send its own timestamp value.
This can be useful if Riemann is inaccessible for a while, and the messages are collected in the disk buffer until Riemann is accessible again. In this case, it would be difficult to differentiate between messages based on the arrival time only, because this would mean that there would be hundreds of messages with the same arrival time. This issue can be solved by using this option.
The event-time() option takes an optional parameter specifying whether the time format is in seconds or microseconds. For example:
In case the parameter is omitted, AxoSyslog defaults to the seconds version. In case the event-time() option is omitted altogether, AxoSyslog defaults to the seconds version with $UNIXTIME.
Note that the time format parameter requires:
riemann-c-client 1.10.0 or newer
In older versions of riemann-c-client, the microseconds option is not available.
In case your distribution does not contain a recent enough version of riemann-c-client and you wish to use microseconds, install a new version from .
If you installed the new version in a custom location (instead of the default one), make sure that you append the directory of the pkg-config file (.pc file) to the environment variable export PKG_CONFIG_PATH=....
After calling configure, you should see the following message in the case of successful installation:
Older versions of Riemann cannot handle microseconds. No error will be indicated, however, the time of the event will be set to the timestamp when the message arrived to Riemann.
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
If an error occurs while sending the messages to the server, AxoSyslog will try to resend every message from the batch. If it does not succeed (you can set the number of retry attempts in the retries() option), AxoSyslog drops every message in the batch.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The value to add as the host field of the Riemann event.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
metric()
Type:
template, macro, or string
Default:
Description: The numeric value to add as the metric field of the Riemann event. If possible, include type-hinting as well, otherwise the Riemann server will interpret the value as a floating-point number. The following example specifies the SEQNUM macro as an integer.
metric(int("$SEQNUM"))
port()
Type:
number
Default:
5555
Description: The port number of the Riemann server.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
server()
Type:
hostname or IP address
Default:
127.0.0.1
Description: The hostname or IP address of the Riemann server.
service()
Type:
template, macro, or string
Default:
${PROGRAM}
Description: The value to add as the service field of the Riemann event.
state()
Type:
template, macro, or string
Default:
Description: The value to add as the state field of the Riemann event.
tags()
Type:
string list
Default:
the tags already assigned to the message
Description: The list of tags to add as the tags field of the Riemann event. If not specified AxoSyslog automatically adds the tags already assigned to the message. If you set the tags() option, only the tags you specify will be added to the event.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
timeout()
Type:
number [seconds]
Default:
Description: The value (in seconds) to wait for an operation to complete, and attempt to reconnect the Riemann server if exceeded. By default, the timeout is disabled.
ttl()
Type:
template, macro, or number
Default:
Description: The value (in seconds) to add as the ttl (time-to-live) field of the Riemann event.
type()
Type:
tcp
Default:
tcp
Description: The type of the network connection to the Riemann server: TCP, TLS, or UDP. For TLS connections, set the ca-file() option to authenticate the Riemann server, and the cert-file() and key-file() options if the Riemann server requires authentication from its clients.
Make sure that you specify TLS options either using type() or using the tls() block. Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
ca-file()
Type:
path to a CA certificate in PEM format
Default:
Description: Path to the CA certificate in PEM format that signed the certificate of the Riemann server. When establishing TLS connection, AxoSyslog verifies the certificate of the Riemann server using this CA.
This option was called cacert() up until (and including) AxoSyslog version 3.12.
cert-file()
Type:
path to a CA certificate in PEM format
Default:
Description: Path to the a certificate file in PEM format. When establishing TLS connection, AxoSyslog authenticates on the Riemann server using this certificate and the matching private key set in the key-file() option.
Note that you have to set the cert-file() and key-file() options only if the Riemann server requires authentication from the clients.
This option was called cert() in AxoSyslog version 3.7.
key-file()
Type:
path to a private key file
Default:
Description: Path to the private key of the certificate file set in the cert-file() option. When establishing TLS connection, AxoSyslog authenticates on the Riemann server using this private key and the matching certificate set in the cert-file() option.
Note that you have to set the cert-file() and key-file() options only if the Riemann server requires authentication from the clients.
This option was called key() in AxoSyslog version 3.7.
8.33 - s3: Amazon S3
Available in AxoSyslog version 4.4 and later.
The s3() destination sends log messages to the Amazon Simple Storage Service (Amazon S3) object storage service. Messages are normally sent encrypted with TLS (HTTPS), but you can specify a custom unencrypted HTTP endpoint.
Prerequisites
An existing S3 bucket configured for programmatic access, and the related ACCESS_KEY and SECRET_KEY of a user that can access it.
If you are not using the venv (/usr/bin/syslog-ng-update-virtualenv) created by AxoSyslog, you must install the boto3 and/or botocore Python dependencies.
To use the s3() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The s3() driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
AxoSyslog can create a new object based on the following strategies:
Based on object size: The max-object-size() option configures AxoSyslog to finish an object if it reaches a certain size. AxoSyslog appends an index ("-1", “-2”, …) to the end of the object key, then starts a new object.
Based on timestamp: The object-key-timestamp() option can be used to set a datetime-related template, which is appended to the end of the object, for example: "${R_MONTH_ABBREV}${R_DAY}". When a log message arrives with a newer timestamp template resolution, the previous timestamped object gets finished and a new one is started with the new timestamp. If an older message arrives, it doesn`t reopen the old object, but starts a new object with the key having an index appended to the old object.
Based on timeout: The flush-grace-period() option sets the number of minutes to wait for new messages to arrive after the last one. If the timeout expires, AxoSyslog closes the object, and opens a new object (with an appended index) when a new message arrives.
All of these strategies can be used individually, or together.
Upload options
AxoSyslog uploads objects using the multipart upload API. AxoSyslog composes chunks locally. When a chunk reaches the size set in chunk-size() (by default 5 MiB), the chunk is uploaded. When an object is finished, the multipart upload is completed and S3 merges the chunks.
The following options are specific to the s3() destination.
access-key()
Type:
string
Default:
N/A
Description: The ACCESS_KEY of the service account used to access the S3 bucket. (Together with secret-key().)
Starting with version 4.7, you can use the AWS_... environment variables or credentials files from the ~/.aws/ directory instead of this option. For details, see the official documentation.
bucket()
Type:
string
Default:
Description: The name of the S3 bucket, for example, my-bucket. Note that the bucket must already exist.
canned-acl()
Type:
string
Default:
empty
Description: The ACL assigned to the object, if specified, for example, bucket-owner-read. The following values are valid:
If you configure an invalid value, the default is used.
chunk-size()
Type:
string
Default:
5MiB
Description: The size of log messages that AxoSyslog writes to the S3 object in a batch. If compression is enabled, the chunk-size() refers to the compressed size.
compression()
Type:
boolean
Default:
no
Description: Setting compression(yes) enables gzip compression, and implicitly adds a .gz suffix to the created object’s key. You can set the level of the compression using the compresslevel() option (0-9).
compresslevel()
Type:
integer (0-9)
Default:
9
Description: Only has effect if compression() is set to yes. You can set the level of the compression using the compresslevel() option (0-9).
flush-grace-period()
Type:
integer [minutes]
Default:
60
Description: After the grace period expires and no new messages are routed to the destination, AxoSyslog flushes the contents of the buffer to the S3 object even if the volume of the messages in the buffer is lower than chunk-size().
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
max-object-size()
Type:
string
Default:
5120GiB
Description: The maximal size of the S3 object. If an object reaches this size, AxoSyslog appends an index ("-1", “-2”, …) to the end of the object key and starts a new object after rotation.
max-pending-uploads()
Type:
integer
Default:
32
Description: The max-pending-uploads() and upload-threads() options configure the upload of the chunks. Uploading happens in multiple threads to minimize network overhead.
upload-threads() limits the maximum number of parallel uploads.
max-pending-uploads() limits the number of chunks that are waiting in the work queue of the upload threads to get uploaded.
object-key()
Type:
template
Default:
N/A
Description: The object key (or key name), which uniquely identifies the object in an Amazon S3 bucket. Note that a suffix may be appended to this object key depending on the naming strategies used. Example: my-logs/${HOSTNAME}/.
object-key-timestamp()
Type:
template
Default:
Description: The object-key-timestamp() option can be used to set a datetime-related template, which is appended to the end of the object key, for example: "${R_MONTH_ABBREV}${R_DAY}". When a log message arrives with a newer timestamp template resolution, the previous timestamped object gets finished and a new one is started with the new timestamp. If an older message arrives, it doesn`t reopen the old object, but starts a new object with the key having an index appended to the old object.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
region()
Type:
string
Default:
Description: The AWS region to use when writing the bucket. This should normally be the same region where the bucket is created. This option implies an API endpoint url(). For providers other than AWS, or for custom API endpoints, use the url() option.
secret-key()
Type:
string
Default:
N/A
Description: The SECRET_KEY of the service account used to access the S3 bucket. (Together with access-key().)
Starting with version 4.7, you can use the AWS_... environment variables or credentials files from the ~/.aws/ directory instead of this option. For details, see the official documentation.
storage-class()
Type:
string
Default:
STANDARD
Description: The storage class of the object, for example, REDUCED_REDUNDANCY. The following values are valid:
If you configure an invalid value, the default is used.
upload-threads()
Type:
integer
Default:
8
Description: The number of AxoSyslog worker threads that are used to upload data to S3 from this destination.
template()
Type:
template or template function
Default:
${MESSAGE}\n
Description: The message as written to the Amazon S3 object. You can use templates and template functions to format the message.
url()
Type:
string
Default:
N/A
Description: The API endpoint URL for writing to the S3 bucket, for example https://s3.us-west-2.amazonaws.com, http://minio.local:9000, or https://storage.googleapis.com.
8.34 - slack: Send alerts and notifications to a Slack channel
The slack() destination driver sends messages to a Slack channel using the Slack Web API. For the list of available optional parameters, see Slack destination options. This destination is available in version 3.19 and later.
The driver allows you to modify nearly every field of the HTTP request. For details, see the Slack API documentation.
You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.
By default, the throttle() option is set to 1, because Slack has a 1 message/second limit on Webhooks. It can allow more message in short bursts, so you can set it to 0, if you only expect messages in a short period of time. For details, see the Web API rate limiting in the Slack documentation.
To use this destination, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The slack() driver is actually a reusable configuration snippet configured to send log messages using the http() driver. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
Prerequisites
To send messages and notifications from AxoSyslog to Slack, you must create a Slack app and a Webhook that AxoSyslog can use. For details, see the Slack documentation.
Example: Using the slack() driver
The following example sets the colors and the author of the message.
The slack destination of AxoSyslog can directly post log messages and notifications to Slack channels. The slack destination has the following options.
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
ca-file()
Accepted values:
Filename
Default:
none
Description: Name of a file that contains an X.509 CA certificate (or a certificate chain) in PEM format. The AxoSyslog application uses this certificate to validate the certificate of the HTTPS server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
cipher-suite()
Accepted values:
Name of a cipher, or a colon-separated list
Default:
Depends on the OpenSSL version that AxoSyslog uses
Description: Specifies the cipher, hash, and key-exchange algorithms used for the encryption, for example, ECDHE-ECDSA-AES256-SHA384. The list of available algorithms depends on the version of OpenSSL used to compile AxoSyslog. To specify multiple ciphers, separate the cipher names with a colon, and enclose the list between double-quotes, for example:
For a list of available algorithms, execute the openssl ciphers -v command. The first column of the output contains the name of the algorithms to use in the cipher-suite() option, the second column specifies which encryption protocol uses the algorithm (for example, TLSv1.2). That way, the cipher-suite() also determines the encryption protocol used in the connection: to disable SSLv3, use an algorithm that is available only in TLSv1.2, and that both the client and the server supports. You can also specify the encryption protocols using ssl-options().
You can also use the following command to automatically list only ciphers permitted in a specific encryption protocol, for example, TLSv1.2:
echo"cipher-suite(\"$(openssl ciphers -v | grep TLSv1.2 | awk '{print $1}'| xargs echo -n | sed 's/ /:/g'| sed -e 's/:$//')\")"
Note that starting with version 3.10, when AxoSyslog receives TLS-encrypted connections, the order of ciphers set on the AxoSyslog server takes precedence over the client settings.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
Description: The colors to be assigned to the messages of different importance levels.
color-chooser()
Type:
integer or template
Default:
‘${LEVEL_NUM}’
Description: An integer that assigns a color to the message from the list of colors set in the colors() option.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The Webhook URL for the Incoming Webhook of your Slack app. This URL must also include the authentication token that AxoSyslog uses to authenticate to Slack. For example: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
Description: The number of messages that the output queue can store.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
To handle HTTP error responses, if the HTTP server returns 5xx codes, AxoSyslog will attempt to resend messages until the number of attempts reaches retries. If the HTTP server returns 4xx codes, AxoSyslog will drop the messages.
ssl-version()
Type:
string
Default:
None, uses the libcurl default
Description: Specifies the permitted SSL/TLS version. Possible values: sslv2, sslv3, tlsv1, tlsv1_0, tlsv1_1, tlsv1_2, tlsv1_3.
An alternative way to specify this option is to put it into a tls() block, together with any other TLS options. This allows you to separate these options and ensure better readability.
Make sure that you specify TLS options either using their own dedicated option (ca-dir(), ca-file(), cert-file(), cipher-suite(), key-file(), peer-verify(), and ssl-version()), or using the tls() block and inserting the relevant options within tls(). Avoid mixing the two methods. In case you do specify TLS options in both ways, the one that comes later in the configuration file will take effect.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
By default, the throttle() option is set to 1, because Slack has a 1 message/second limit on Webhooks. It can allow more message in short bursts, so you can set it to 0, if you only expect messages in a short period of time. For details, see the Web API rate limiting in the Slack documentation.
Description: The value (in seconds) to wait for an operation to complete, and attempt to reconnect the server if exceeded. By default, the timeout value is 0, meaning that there is no timeout. Available in version 3.11 and later.
Description: The value of the USER-AGENT header in the messages sent to the server.
use-system-cert-store()
Type:
`yes
Default:
no
Description: Use the certificate store of the system for verifying HTTPS certificates. For details, see the curl documentation.
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url("site1", "site2", "site3")), set the workers() option to 3 or more.
8.35 - smtp: Generate SMTP messages (emails) from logs
The destination is aimed at a fully controlled local, or near-local, trusted SMTP server. The goal is to send mail to trusted recipients, through a controlled channel. It hands mails over to an SMTP server, and that is all it does, therefore the resulting solution is as reliable as sending an email in general. For example, AxoSyslog does not verify whether the recipient exists.
The smtp() driver sends email messages triggered by log messages. The smtp() driver uses SMTP, without needing external applications. You can customize the main fields of the email, add extra headers, send the email to multiple recipients, and so on.
The subject(), body(), and header() fields may include macros which get expanded in the email. For more information on available macros see Macros of AxoSyslog.
The smtp() driver has the following required parameters: host(), port(), from(), to(), subject(), and body(). For the list of available optional parameters, see smtp() destination options.
Note
The smtp() destination driver is available only in AxoSyslog 3.4 and later.
The following example defines an smtp() destination using only the required parameters.
destination d_smtp { smtp( host("localhost") port(25) from("alert service""[email protected]") to("Admin #1""[email protected]") subject("[ALERT] Important log message of $LEVEL condition received from $HOST/$PROGRAM!") body("Hi!\nThe alerting service detected the following important log message:\n $MSG\n-- \n"));};
The following example sets some optional parameters as well.
destination d_smtp { smtp( host("localhost") port(25) from("syslog-ng alert service""[email protected]") to("Admin #1""[email protected]") to("Admin #2""[email protected]") cc("Admin BOSS""[email protected]") bcc("Blind CC""[email protected]") subject("[ALERT] Important log message of $LEVEL condition received from $HOST/$PROGRAM!") body("Hi!\nThe syslog-ng alerting service detected the following important log message:\n $MSG\n-- \nsyslog-ng\n") header("X-Program", "$PROGRAM"));};
Example: Simple email alerting with the smtp() driver
The following example sends an email alert if the eth0 network interface of the host is down.
filter f_linkdown { match("eth0: link down" value("MESSAGE"));}; destination d_alert { smtp( host("localhost") port(25) from("syslog-ng alert service""syslog@localhost") reply-to("Admins""root@localhost") to("Ennekem""me@localhost") subject("[SYSLOG ALERT]: eth0 link down") body("Syslog received an alert:\n$MSG"));}; log { source(s_local); filter(f_linkdown); destination(d_alert);};
8.35.1 - smtp() destination options
The smtp() sends email messages using SMTP, without needing external applications. The smtp() destination has the following options:
body()
Type:
string
Default:
n/a
Description: The BODY field of the email. You can also use macros in the string. Use \\n to start a new line. For example:
body("AxoSyslog received the following alert from $HOST:\n$MSG")
bcc()
Type:
string
Default:
n/a
Description: The BCC recipient of the email (contents of the BCC field). You can specify the email address, or the name and the email address. Set the bcc option multiple times to send the email to multiple recipients. For example: bcc("[email protected]") or bcc("Admin" "[email protected]") or >bcc("Admin" "[email protected]") bcc("Admin2" "[email protected]")
You can also use macros to set the value of this parameter.
cc()
Type:
string
Default:
n/a
Description: The CC recipient of the email (contents of the CC field). You can specify the email address, or the name and the email address. Set the cc option multiple times to send the email to multiple recipients. For example: cc("[email protected]") or cc("Admin" "[email protected]") or cc("Admin" "[email protected]") cc("Admin2" "[email protected]"
You can also use macros to set the value of this parameter.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
from()
Type:
string
Default:
n/a
Description: The sender of the email (contents of the FROM field). You can specify the email address, or the name and the email address. For example:
If you specify the from() option multiple times, the last value will be used. Instead of the from() option, you can also use sender(), which is just an alias of the from() option.
You can also use macros to set the value of this parameter.
header()
Type:
string
Default:
n/a
Description: Adds an extra header to the email with the specified name and content. The first parameter sets the name of the header, the second one its value. The value of the header can contain macros. Set the header option multiple times to add multiple headers. For example:
header("X-Program", "$PROGRAM")
When using the header option, note the following points:
Do not use the header() option to set the values of headers that have dedicated options. Use it only to add extra headers.
If you set the same custom header multiple times, only the first will be added to the email, other occurrences will be ignored.
It is not possible to set the DATE, Return-Path, Original-Recipient, Content-*, MIME-*, Resent-*, Received headers.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Hostname or IP address of the SMTP server.
Note
If you specify host="localhost", AxoSyslog will use a socket to connect to the local SMTP server. Use host="127.0.0.1" to force TCP communication between AxoSyslog and the local SMTP server.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
port()
Type:
number
Default:
25
Description: The port number of the SMTP server.
reply-to()
Type:
string
Default:
n/a
Description: Replies of the recipient will be sent to this address (contents of the REPLY-TO field). You can specify the email address, or the name and the email address. Set the reply-to() option multiple times to send the email to multiple recipients. For example: reply-to("[email protected]") or reply-to("Admin" "[email protected]") or reply-to("Admin" "[email protected]") reply-to("Admin2" "[email protected]")
You can also use macros to set the value of this parameter.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
subject()
Type:
string
Default:
n/a
Description: The SUBJECT field of the email. You can also use macros. For example:
subject("[SYSLOG ALERT]: Critical error message received from $HOST")
If you specify the subject() option multiple times, the last value will be used.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
to()
Type:
string
Default:
localhost
Description: The recipient of the email (contents of the TO field). You can specify the email address, or the name and the email address. Set the to() option multiple times to send the email to multiple recipients. For example: to("[email protected]") or to("Admin" "[email protected]") or to("Admin" "[email protected]") to("Admin2" "[email protected]")
You can also use macros to set the value of this parameter.
8.36 - snmp: Send SNMP traps
The snmp() driver sends SNMP traps using the Simple Network Management Protocol version 2c or version 3. Incoming log messages can be converted to SNMP traps, as the fields of the SNMP messages can be customized using AxoSyslog macros.
The snmp() driver is available in AxoSyslog version 3.22 and later.
Note
The snmp destination driver currently supports sending SNMP traps only using the UDP transport protocol.
The snmp() driver requires the host(), trap-obj(), and snmp-obj() options to be set, as well as the engine-id() and version() options when using the SNMPv3 protocol. For the list of available optional parameters, see snmp() destination options.
If AxoSyslog cannot resolve the destination hostname during startup, it will try to resolve the hostname again when the next message to be sent as an SNMP trap is received. However, if this name resolution fails, the trap will be dropped.
Note
The snmp() destination driver does not generate MARK signals itself, but can receive and forward MARK signals.
Example: Using the snmp() destination driver
The following example defines an SNMP destination that uses the SNMPv2c protocol.
8.36.1 - Converting Cisco syslog messages to clogMessageGenerated SNMP traps
The AxoSyslog application can convert the syslog messages sent by Cisco devices to Cisco-specific SNMP traps defined by the CISCO-SYSLOG-MIB (enterprises.cisco.ciscoMgmt.ciscoCiscoMIB) is also supported (such traps are also referred to as clogMessageGenerated notifications). That way, the incoming log messages can be forwarded to devices used to process and analyze Cisco-specific SNMP traps. For this to work correctly, the following requirements must be met:
The Source Configuration Library (SCL) must be included in the AxoSyslog configuration file:
To accomplish this, AxoSyslog has to use a special pattern database to parse the Cisco-specific syslog messages, because these messages do not comply with the standard syslog formats.
For details on the Cisco-specific SNMP trap format, see CISCO-SYSLOG-MIB on the Cisco website.
Parsing Cisco-specific message fields with patterndb
The ${PROGRAM} part of the syslog messages sent by Cisco devices contain not only the program name, but other important protocol information part as well. The ${PROGRAM} of these messages contains the Facility, Severity, and the Mnemonic (the Cisco name) of the message. The following pattern database parses these values and makes them available as the .cisco.Facility, .cisco.Severity, and .cisco.MsgName, respectively. The actual log message is available as .cisco.MsgText.
<patterndb version="4"pub_date="2011-05-03">
<ruleset name="cisco snmp ruleset1" xml:id="480de478-d4a6-4a7f-bea4-0c0245d361e3">
<description>Pattern for Cisco messages having BSD timestamps, for example: Jul 012010 00:32:59: %SYS-5-CONFIG_I: Configured from console by console</description>
<pattern>%@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ANYSTRING:.cisco.MsgName@</pattern>
<rules>
<rule xml:id="09944c71-95eb-4bc0-8575-936931d85713"provider="oneidentity"class="system">
<patterns>
<pattern> @ANYSTRING:.cisco.MsgText@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
<ruleset name="cisco snmp ruleset2" xml:id="480de478-d4a6-4a7f-bea4-0c0245d361e3">
<description>Pattern for Cisco messages having cisco-specific timestamps, for example: 18: Jan 22 10:45:44.543: %SYS-5-CONFIG_I: Configured from console by console</description>
<rules>
<rule xml:id="09944c71-95eb-4bc0-8575-936931d85714"provider="oneidentity"class="system">
<patterns>
<pattern>%@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ESTRING:.cisco.MsgName::@ @ANYSTRING:.cisco.MsgText@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>
Sending clogMessageGenerated SNMP traps
To send out clogMessageGenerated SNMP traps, use the cisco_snmp() destination driver. The cisco-snmp() destination is actually a modified version of the snmp() destination driver.
Note
The cisco-snmp() driver is actually a reusable configuration snippet. For details on using or writing SCLs, see Reusing configuration blocks.
The cisco-snmp() driver has the same requirements and options as the snmp() destination driver, but automatically fills the clogMessageGenerated-specific fields with the data received from parsing the Cisco-specific syslog messages using the pattern database. For details on the , see the <INSTALLDIR>/ share/include/scl/snmp/plugin.conf file.
This driver sends SNMP traps using the SNMP v2c or v3 protocol.
The snmp() destination has the following options:
auth-algorithm()
Type:
SHA
Default:
SHA
Description: The authentication method to use. Lowercase values (for example, sha) can be used as well.
This option is used with the SNMPv3 protocol.
auth-password()
Type:
string
Default:
empty string
Description: The password used for authentication. If the auth-username() option is set but the auth-password() is empty, AxoSyslog will try to authenticate with an empty password.
This option is used with the SNMPv3 protocol.
auth-username()
Type:
string
Default:
empty string
Description: The username used to authenticate on the SNMP server. If this parameter is set, AxoSyslog will try to authenticate on the SNMP server.
This option is used with the SNMPv3 protocol.
community()
Type:
string
Default:
public
Description: The community string used for SNMPv2c authentication.
This option is used with the SNMPv2c protocol.
enc-algorithm()
Type:
AES
Default:
AES
Description: The encryption method used to encrypt the SNMP traffic. Lowercase values (for example, aes) can be used as well.
This option is used with the SNMPv3 protocol.
enc-password()
Type:
string
Default:
Description: The password used for the encryption. Encryption is used only if the enc-password() is not empty.
This option is used with the SNMPv3 protocol.
engine-id()
Type:
number (hexadecimal number)
Default:
Description: The engine ID is a hexadecimal number at least 10 digits long, starting with 0x. for example, 0xABABABABAB.
This option is a required parameter when using the SNMPv3 protocol.
host()
Type:
hostname or IP address
Default:
n/a
Description: Hostname of the SNMP server.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
Description: The snmp-obj() option can be used to create custom SNMP trap elements. To create a trap element, specify the OID, type, and value of the element in the snmp-obj() option. To send SNMP traps, at least one snmp-obj() option must be defined. The snmp-obj() option requires the following parameters. Note that AxoSyslog does not validate the values of these elements.
<oid_of_the_object>: The object id of the SNMP object, for example, .1.3.6.1.4.1.18372.3.1.1.1.1.1`.
<type_of_the_object>: The type of the object specified as an ASN.1 primitive. One of: Integer, Timeticks, Octetstring, Counter32, Ipaddress, Objectid`. The type names are not case sensitive.
<value_of_the_object>: The value of the object as a string. The macros of AxoSyslog can be used to set these values, making it possible to transfer the content and other metadata from the the syslog message to the SNMP trap. Note that if the value of an Integer, Counter32orTimeticks` object is not a number (for example, is an empty string or other not-number string), AxoSyslog will automatically replace the value with 0. The values of other types of objects are not validated.
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
Description: The trap-obj() is a specialized version of the snmp-obj() option that is used to identify the SNMP trap object. The type of the trap object is always Objectid. The <oid_of_the_object> and the <value_of_the_object»parameters are identical to the respective parameters of the snmp-obj() option. For details on these parameters, see snmp-obj().
Note
Using the trap-obj() object is equivalent to using the snmp-obj() with the Objectid type.
version()
Type:
v2c
Default:
v2c
Description: Specifies which version of the SNMP protocol to use.
Note
The AxoSyslog application will accept any valid option for the snmp() destination, but will only use the ones relevant to the selected protocol version, any other option will be ignored. For example, if the version("v2c") engine-id("0xABABABABAB") community("mycommunity") options are set, AxoSyslog will accept every option, but process only the community() option, because engine-id() applies only to SNMPv3.
8.37 - splunk-hec-event: Send messages to Splunk HEC
Starting with version 4.2.0, AxoSyslog can send messages to the Splunk HTTP Event Collector (HEC).
Prerequisites
Enable the HTTP Event Collector (HEC) on your Splunk deployment.
Create a token for AxoSyslog to use in the token() option of the destination. When creating the token, use the syslog source type.
event() accepts a template, which declares the content of the log message sent to Splunk. Default value: ${MSG}
index(), source(), host(), and time() accept templates, and declare the respective field of each log message based on the set template.
default-index(), default-source(), and default-sourcetype() accept literal strings, and are used as fallback values if a log message doesn’t set these fields. These values are passed to the URL as query parameters, so they don’t inflate the body of the HTTP request
for each message in the batch, which saves bandwidth.
fields() accepts template, which is passed as additional indexing metadata to Splunk.
extra-headers(), extra-queries(), and content-type() are additional HTTP request options.
HEC raw API
The splunk-hec-raw() destination feeds Splunk via the HEC raw API.
The options of the splunk-hec-raw() destination are similar to the splunk-hec-event() destination, but it has a mandatory option: channel(). The channel() option must be a globally unique channel identifier (GUID), this ID differentiates the data from different clients. Note that Splunk doesn’t generate this ID, you must create it for yourself. When Splunk sees a new channel identifier, it creates a new channel.
Use the template() option to set the content of the log message sent to Splunk (and not the event() option that is used in the splunk-hec-event() destination).
8.38 - sql: Store messages in an SQL database
The sql() driver sends messages into an SQL database. Currently the Microsoft SQL (MSSQL), MySQL, Oracle, PostgreSQL, and SQLite databases are supported. Starting with AxoSyslog 4.0, type information is automatically added to the stored columns if available. For details, see Specifying data types in value-pairs.
The AxoSyslog application requires read and write access to the SQL table, otherwise it cannot verify that the destination table exists.
Currently the AxoSyslog application has default schemas for the different databases and uses these defaults if the database schema (for example, columns and column types) is not defined in the configuration file. However, these schemas will be deprecated and specifying the exact database schema will be required in later versions.
Note
The sql() destination requires database-specific packages to be installed. These packages are automatically installed by the binary syslog-ng installer.
The table and value parameters can include macros to create tables and columns dynamically (for details, see Macros of AxoSyslog).
Warning
When using macros in table names, note that some databases limit the maximum allowed length of table names. Consult the documentation of the database for details.
Inserting the records into the database is performed by a separate thread. The AxoSyslog application automatically performs the escaping required to insert the messages into the database.
Example: Using the sql() driver
The following example sends the log messages into a PostgreSQL database running on the logserver host. The messages are inserted into the logs database, the name of the table includes the exact date and the name of the host sending the messages. The AxoSyslog application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.
8.38.1 - Using the sql() driver with an Oracle database
The Oracle sql destination has some special aspects that are important to note.
The hostname of the database server is set in the tnsnames.ora file, not in the host parameter of the sql() destination.
If the tnsnames.ora file is not located in the /etc directory (or in the /var/opt/oracle directory on Solaris), set the following Oracle-related environment variables, so AxoSyslog will find the file: ORACLE_BASE, ORACLE_HOME, and ORACLE_SID. For details, see the documentation of the Oracle Instant Client.
You cannot use the same database() settings in more than one destination, because the database() option of the SQL driver is just a reference to the connection string of the tnsnames.ora file. To overcome this problem, you can duplicate the connections in the tnsnames.ora file under a different name, and use a different table in each Oracle destination in AxoSyslog.
As certain database versions limit the maximum length of table names, macros in the table names should be used with care.
In the current version of AxoSyslog, the types of database columns must be explicitly set for the Oracle destination. The column used to store the text part of the syslog messages should be able to store messages as long as the longest message permitted by syslog-ng, therefore it is usually recommended to use the varchar2 or clob column type. (The maximum length of the messages can be set using the log-msg-size() option.) For details, see the following example.
The Oracle Instant Client used by AxoSyslog supports only the following character sets:
Single-byte character sets: US7ASCII, WE8DEC, WE8MSWIN1252, and WE8ISO8859P1
Unicode character sets: UTF8, AL16UTF16, and AL32UTF8
Example: Using the sql() driver with an Oracle database
The following example sends the log messages into an Oracle database running on the logserver host, which must be set in the /etc/tnsnames.ora file. The messages are inserted into the LOGS database, the name of the table includes the exact date when the messages were sent. The AxoSyslog application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.
The Oracle Instant Client retrieves the address of the database server from the /etc/tnsnames.ora file. Edit or create this file as needed for your configuration. A sample is provided below.
8.38.2 - Using the sql() driver with a Microsoft SQL database
The mssql database driver can access Microsoft SQL (MSSQL) destinations. This driver has some special aspects that are important to note.
The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the AxoSyslog server. For details, see the following example.
As certain database versions limit the maximum length of table names, macros in the table names should be used with care.
In the current version of AxoSyslog, the types of database columns must be explicitly set for the MSSQL destination.
Warning
The following column types cannot be used in MSSQL destinations: nchar, nvarchar, ntext, and xml.
The column used to store the text part of the syslog messages should be able to store messages as long as the longest message permitted by syslog-ng. The varchar column type can store maximum 4096 bytes-long messages. The maximum length of the messages can be set using the log-msg-size() option. For details, see the following example.
Remote access for SQL users must be explicitly enabled on the Microsoft Windows host running the Microsoft SQL Server.
Example: Using the sql() driver with an MSSQL database
The following example sends the log messages into an MSSQL database running on the logserver host. The messages are inserted into the syslogng database, the name of the table includes the exact date when the messages were sent. The AxoSyslog application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.
The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the AxoSyslog server. Edit or create this file as needed for your configuration. A sample is provided below.
[default]date="%Y-%m-%d %H:%M:%S"
8.38.3 - Interacting with the database
SQL operations
Create table:
If the given table does not exist, AxoSyslog tries to create it with the given column types.
The AxoSyslog application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.
If AxoSyslog cannot create or alter a table, it tries to do it again when it reaches the next time-reopen().
Alter table:
If the table structure is different from given structure in an existing table, AxoSyslog tries to add columns in this table but never will delete or modify an existing column.
If AxoSyslog cannot create or alter a table, it tries to do it again when reach the next time-reopen().
The AxoSyslog application requires read and write access to the SQL table, otherwise it cannot verify that the destination table exists.
Insert table:
Insert new records in a table.
Inserting the records into the database is performed by a separate thread.
The AxoSyslog application automatically performs the escaping required to insert the messages into the database.
If insert returns with error, AxoSyslog tries to insert the message +two times by default, then drops it. Retrying time is the value of time-reopen().
Encoding
The AxoSyslog application uses UTF-8 by default when writes logs into database.
Start/stop and reload
Start:
The AxoSyslog application will connect to database automatically after starting regardless existing incoming messages.
Stop:
The AxoSyslog application will close the connection to database before shutting down.
Possibility of losing logs:
The AxoSyslog application cannot lose logs during shutting down if disk buffer was given and it is not full yet.
The AxoSyslog application cannot lose logs during shutting down if disk buffer was not given.
Reload:
The AxoSyslog application will close the connection to database if it received SIGHUP signal (reload).
It will reconnect to the database when it tries to send a new message to this database again.
Macros:
The value of ${SEQNUM} macro will be overridden by sql driver regardless of local or relayed incoming message.
It will be grown continuously.
8.38.3.1 - MySQL-specific interaction methods
To specify the socket to use, set and export the MYSQL_UNIX_PORT environment variable, for example, MYSQL_UNIX_PORT=/var/lib/mysql/mysql.sock; export MYSQL_UNIX_PORT.
8.38.3.2 - MsSQL-specific interaction methods
In SQL Server 2005 this restriction is lifted - kind of. The total length of all key columns in an index cannot exceed 900 bytes.
If you are using null() in your configuration, be sure that the columns allow NULL to insert. Give the column as the following example: "datetime varchar(16) NULL".
The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the AxoSyslog server. [default] date = "%Y-%m-%d %H:%M:%S".
8.38.4 - sql() destination options
This driver sends messages into an SQL database. The sql() destination has the following options:
batch-bytes()
Accepted values:
number [bytes]
Default:
none
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
Description: Name of the columns storing the data in fieldname [dbtype] format. The [dbtype] parameter is optional, and specifies the type of the field. By default, AxoSyslog creates text columns. Note that not every database engine can index text fields.
Warning
The following column types cannot be used in MSSQL destinations: nchar, nvarchar, ntext, and xml.
create-statement-append()
Type:
string
Default:
empty string
Description: Specifies additional SQL options that are appended to the CREATE statement. That way you can customize what happens when AxoSyslog creates a new table in the database. Consult the documentation of your database server for details on the available options. Syntax:
create-statement-append(<options-to-append>)
For example, you can appends the ROW_FORMAT=COMPRESSED option to MySQL create table statements:
create-statement-append(ROW_FORMAT=COMPRESSED)
database()
Type:
string
Default:
logs
Description: Name of the database that stores the logs. Macros cannot be used in database name. Also, when using an Oracle database, you cannot use the same database() settings in more than one destination.
dbd-option()
Type:
string
Default:
empty string
Description: Specify database options that are set whenever AxoSyslog connects to the database server. Consult the documentation of your database server for details on the available options. Syntax:
dbd-option(OPTION_NAME VALUE)
OPTION_NAME is always a string, VALUE is a string or a number. For example:
Description: Defines an optional DBI driver location for DBD initialization.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Flags related to the sql() destination.
dont-create-tables: Enable this flag to prevent AxoSyslog from creating non-existing database tables automatically. The AxoSyslog application typically has to create tables if you use macros in the table names. Available in AxoSyslog version 3.2 and later.
explicit-commits: By default, AxoSyslog commits every log message to the target database individually. When the explicit-commits option is enabled, messages are committed in batches. This improves the performance, but results in some latency, as the messages are not immediately sent to the database. The size and frequency of batched commits can be set using the batch-lines() parameter. The explicit-commits option is available in AxoSyslog version 3.2 and later.
Example: Setting flags for SQL destinations
The following example sets the dont-create-tables and explicit-commits flags for an sql() destination.
flags(dont-create-tables,explicit-commits)
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Hostname of the database server. Note that Oracle destinations do not use this parameter, but retrieve the hostname from the /etc/tnsnames.ora file.
Note
If you specify host="localhost", AxoSyslog will use a socket to connect to the local database server. Use host="127.0.0.1" to force TCP communication between AxoSyslog and the local database server.
To specify the socket to use, set and export the MYSQL_UNIX_PORT environment variable, for example, MYSQL_UNIX_PORT=/var/lib/mysql/mysql.sock; export MYSQL_UNIX_PORT.
indexes()
Type:
string list
Default:
“date”, “facility”, “host”, “program”
Description: The list of columns that are indexed by the database to speed up searching. To disable indexing for the destination, include the empty indexes() parameter in the destination, simply omitting the indexes parameter will cause AxoSyslog to request indexing on the default columns.
The AxoSyslog application will create the name of indexes automaticaly with the following method:
In case of MsSQL, PostgreSQL, MySQL or SQLite or (Oracle but tablename < 30 characters): {table}_{column}_idx.
In case of Oracle and tablename > 30 characters: md5sum of {table}_{column}-1 and the first character will be replaced by “i” character and the md5sum will be truncated to 30 characters.
local-time-zone()
Type:
name of the timezone, or the timezone offset
Default:
The local timezone.
Description: Sets the timezone used when expanding filename and tablename templates.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
null()
Type:
string
Default:
Description: If the content of a column matches the string specified in the null() parameter, the contents of the column will be replaced with an SQL NULL value. If unset (by default), the option does not match on any string. For details, see the Example: Using SQL NULL values.
Example: Using SQL NULL values
The null() parameter of the SQL driver can be used to replace the contents of a column with a special SQL NULL value. To replace every column that contains an empty string with NULL, use the null("") option, for example
To replace only a specific column (for example, pid) if it is empty, assign a default value to the column, and use this default value in the null() parameter:
Ensure that the default value you use does not appear in the actual log messages, because other occurrences of this string will be replaced with NULL as well.
password()
Type:
string
Default:
n/a
Description: Password of the database user.
port()
Type:
number
Default:
1433 TCP for MSSQL, 3306 TCP for MySQL, 1521 for Oracle, and 5432 TCP for PostgreSQL
Description: The port number to connect to.
quote-char()
Type:
string
Default:
Available in AxoSyslog version 4.3.0 and newer.
Description: Set custom quoting for table and index names (for example, MySQL needs sometimes this for certain identifiers).
Note: Using a backtick character needs special formatting, because AxoSyslog uses backticks for configuration parameter names. To use backticks as quote character, set a double backtick: quote-char("``")
retries()
Type:
number (insertion attempts)
Default:
3
Description: The number of insertion attempts. If AxoSyslog could not insert a message into the database, it will repeat the attempt until the number of attempts reaches retries, then drops the connection to the database. For example, AxoSyslog will try to insert a message maximum three times by default (once for first insertion and twice if the first insertion was failed).
session-statements()
Type:
comma-separated list of SQL statements
Default:
empty string
Description: Specifies one or more SQL-like statement which is executed after AxoSyslog has successfully connected to the database. For example:
The AxoSyslog application does not validate or limit the contents of customized queries. Consequently, queries performed with a user with write-access can potentially modify or even harm the database. Use customized queries with care, and only for your own responsibility.
table()
Type:
string
Default:
messages
Description: Name of the database table to use (can include macros). When using macros, note that some databases limit the length of table names.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
type()
Type:
mssql, mysql, oracle, pgsql, or sqlite3
Default:
mysql
Description: Specifies the type of the database, that is, the DBI database driver to use. Use the mssql option to send logs to an MSSQL database. For details, see the examples of the databases on the following sections.
Description: The parts of the message to store in the fields specified in the columns() parameter.
It is possible to give a special value calling: default (without quotation marks).It means that the value will be used that is the default of the column type of this value.
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
flush-lines()
Type:
number
Default:
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
flush-timeout()
Type:
time in milliseconds
Default:
10000 [milliseconds]
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends flushes to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every flush-timeout() seconds.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
local-time-zone()
Type:
name of the timezone, or the timezone offset
Default:
The local timezone.
Description: Sets the timezone used when expanding filename and tablename templates.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
pad-size()
Type:
number
Default:
0
Description: If set, AxoSyslog will pad output messages to the specified size (in bytes). Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes).
Warning
Hazard of data loss! If the size of the incoming message is larger than the previously set pad-size() value, AxoSyslog will truncate the message to the specified size. Therefore, all message content above that size will be lost.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
send-time-zone()
Accepted values:
name of the timezone, or the timezone offset
Default:
local timezone
Description: Specifies the time zone associated with the messages sent by syslog-ng, if not specified otherwise in the message or in the destination driver. For details, see Timezones and daylight saving.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
8.40 - stomp: Publish messages using STOMP
The stomp() driver sends messages to servers (message brokers) using the Simple (or Streaming) Text Oriented Message Protocol (STOMP), formerly known as TTMP. AxoSyslog supports version 1.0 of the STOMP protocol. The AxoSyslog stomp() driver supports persistence.
The name-value pairs selected with the value-pairs() option will be sent as STOMP headers, while the body of the STOMP message is empty by default (but you can add custom content using the body() option). Publishing the name-value pairs as headers makes it possible to use the Headers exchange-type and subscribe only to interesting log streams.
The following example shows the default values of the available options.
destination d_stomp { stomp( host("localhost") port(61613) destination("/topic/syslog") body("")# optional, empty by default persistent(yes) ack(no) username("user")# optional, empty by default password("password")# optional, empty by default value-pairs(scope(selected-macros, nv-pairs, sdata)));};
8.40.1 - stomp() destination options
The stomp() driver publishes messages using the Simple (or Streaming) Text Oriented Message Protocol (STOMP).
The stomp() destination has the following options:
ack()
Type:
yes
Default:
no
Description: Request the STOMP server to acknowledge the receipt of the messages. If you enable this option, then after sending a message, AxoSyslog waits until the server confirms that it has received the message. This delay can seriously limit the performance of AxoSyslog if the message rate is high, and the server cannot acknowledge the messages fast enough.
body()
Type:
string
Default:
empty string
Description: The body of the STOMP message. You can also use macros and templates.
destination()
Type:
string
Default:
/topic/syslog
Description: The name of the destination (message queue) on the STOMP server. It can include macros and templates.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The hostname or IP address of the STOMP server.
password()
Type:
string
Default:
n/a
Description: The password used to authenticate on the STOMP server.
persistent()
Type:
yes
Default:
yes
Description: If this option is enabled, the STOMP server or broker will store the messages on its hard disk. That way, the messages will be retained if the STOMP server is restarted, if the message queue is set to be durable on the STOMP server.
port()
Type:
number
Default:
61613
Description: The port number of the STOMP server.
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
username()
Type:
string
Default:
empty string
Description: The username used to authenticate on the STOMP server.
value-pairs()
Type:
parameter list of the value-pairs() option
Default:
scope("selected-macros" "nv-pairs")
Description: The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Structuring macros, metadata, and other value-pairs.
Note
Empty keys are not logged.
8.41 - Sumo Logic destinations: sumologic-http() and sumologic-syslog()
From version 3.27.1, the AxoSyslog application can send log messages to Sumo Logic, a cloud-based log management and security analytics service, by using the sumologic-http() and sumologic-syslog() destinations.
Prerequisites
Currently, using the sumologic-http() and sumologic-syslog() destinations with AxoSyslog has the following prerequisites:
A Sumo Logic account.
If you do not yet have a Sumo Logic account, visit the official Sumo Logic website, and click Start free trial to create an account.
Note
A free trial version of the Sumo Logic account has limited functionalities and is only available for 90 days.
A verified connection and client configuration with the Sumo Logic service.
Warning
To avoid potential data loss, we strongly recommend that you verify your connection and client configuration with the Sumo Logic service before you start using the sumologic-http() or sumologic-syslog() destination with AxoSyslog in a production environment.
(Optional) For using the sumologic-http() destination, you need a HTTP Hosted Collector configured in the Sumo Logic service.
(Optional) For using the sumologic-http() destination, you need the unique HTTP collector code you receive while configuring your Host Collector for HTTP requests.
Limitations
Currently, using the sumologic-syslog() and sumologic-http() destinations with AxoSyslog has the following limitations:
The minimum required version of AxoSyslog is version 3.27.1.
Message format must be in RFC 5424-compliant form. Messages over 64KB in length are truncated.
For more information about the message format limitations, see the Message format section on the official Sumo Logic website.
64 characters long Sumo Logic tokens must be passed in the message body.
Note
Although RFC 5424 limits the structured data field (SD-ID) to 32 characters, Sumo Logic tokens are 64 characters long. If your logging client enforces the 32 characters length limit, you must pass the token in the message body.
To use the sumologic() driver, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
Note
The sumologic() driver is actually a reusable configuration snippet configured to send log messages using the network() and http() destination by using a template. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
8.41.1 - sumologic-http()
The sumologic-http() and sumologic-syslog() destinations send log messages to Sumo Logic, a cloud-based log management and security analytics service.
In addition, the sumologic-http() destination also has the following options.
batch-bytes()
Accepted values:
number [bytes]
Default:
none
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
Available in AxoSyslog version 3.19 and later.
batch-lines()
Type:
number
Default:
1
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
Description: Custom HTTP headers to include in the request, for example, headers("HEADER1: header1", "HEADER2: header2"). If not set, only the default headers are included, but no custom headers.
The following headers are included by default:
X-Syslog-Host:
X-Syslog-Program:
X-Syslog-Facility:
X-Syslog-Level: <loglevel/priority>
Note
The headers() option is a required option for the sumologic-http() destination.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
In version 4.0 and newer, using the tls() option is optional, and Sumo Logic servers are verified using the system certificate store by default. In earlier versions, this was a required option.
In addition, the sumologic-syslog() destination also has the following options.
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
Description: Optional. This option sets the port number of the Sumo Logic server to connect to.
tag()
Type:
string list
Default:
“tag”
Description: Optional. This option specifies the list of tags to add as the tags fields of Sumo Logic messages. If not specified, AxoSyslog automatically adds the tags already assigned to the message. If you set the tag() option, only the tags you specify will be added to the messages.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
In version 4.0 and newer, using the tls() option is optional, and Sumo Logic servers are verified using the system certificate store by default. In earlier versions, this was a required option.
token()
Type:
string
Default:
Description: Required option. The Cloud Syslog Cloud Token that you received from the Sumo Logic service while configuring your cloud syslog source.
8.42 - syslog: Send messages to a remote logserver using the IETF-syslog protocol
The syslog() driver sends messages to a remote host (for example, a syslog-ng server or relay) on the local intranet or internet using the new standard syslog protocol developed by IETF (for details about the new protocol, see IETF-syslog messages). The protocol supports sending messages using the UDP, TCP, or the encrypted TLS networking protocols.
The required arguments of the driver are the address of the destination host (where messages should be sent). The transport method (networking protocol) is optional, syslog-ng uses the TCP protocol by default. For the list of available optional parameters, see syslog() destination options.
Declaration:
syslog(host transport [options]);
Note
Note that the syslog destination driver has required parameters, while the source driver defaults to the local bind address, and every parameter is optional.
The udp transport method automatically sends multicast packets if a multicast destination address is specified. The tcp and tls methods do not support multicasting.
Note
The default ports for the different transport protocols are as follows: UDP — 514, TCP — 514, TLS — 6514.
If a message uses the IETF-syslog format (RFC5424), only the text of the message can be customized (that is, the $MESSAGE part of the log), the structure of the header is fixed.
8.42.1 - syslog() destination options
The syslog() driver sends messages to a remote host (for example, an AxoSyslog server or relay) on the local intranet or internet using the RFC5424 syslog protocol developed by IETF (for details about the protocol, see IETF-syslog messages). The protocol supports sending messages using the UDP, TCP, or the encrypted TLS networking protocols.
These destinations have the following options:
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
close-on-input()
Type:
yes
Default:
yes
Description: By default, AxoSyslog closes destination sockets if it receives any input from the socket (for example, a reply). If this option is set to no, AxoSyslog just ignores the input, but does not close the socket.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the <code>unset()</code> rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a <code>LogMessage</code> will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop <code>unset</code> values, making the <code>LogMessage</code> representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
<p>When creating a new <code>dir()</code> option for a disk buffer, or modifying an existing one, make sure you delete the persist file.</p>
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the <code>dir()</code> path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of <code>reliable()</code> option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Axoflow does not recommend you to change <code>truncate-size-ratio()</code>. Only change its value if you understand the performance implications of doing so.
Example: Examples for using disk-buffer()
In the following case reliable disk-buffer() is used.
Description: Available only in AxoSyslog version 3.17 and later. For details about how client-side failover works, see Client-side failover.
servers()
Type:
list of IP addresses and fully-qualified domain names
Default:
empty
Description: Specifies a secondary destination server where log messages are sent if the primary server becomes inaccessible. To list several failover servers, separate the address of the servers with comma. By default, AxoSyslog waits for the a server before switching to the next failover server is set in the time-reopen() option.
If failback() is not set, AxoSyslog does not attempt to return to the primary server even if it becomes available. In case the failover server fails, AxoSyslog attempts to connect the next failover server in the list in round-robin fashion.
Warning
The failover servers must be accessible on the same port as the primary server.
failback()
Description: Available only in AxoSyslog version 3.17 and later.
When AxoSyslog starts up, it always connects to the primary server first. In the failover() option there is a possibility to customize the failover modes.
Depending on how you set the failback() option, AxoSyslog behaves as follows:
round-robin mode: If failback() is not set, AxoSyslog does not attempt to return to the primary server even if it becomes available. In case the failover server fails, AxoSyslog attempts to connect the next failover server in the list in round-robin fashion.
In the following example AxoSyslog handles the logservers in round-robin fashion if the primary logserver becomes inaccessible (therefore failback() option is not set).
failback mode: If failback() is set, AxoSyslog attempts to return to the primary server.
After AxoSyslog connects a secondary server during a failover, it sends a probe every tcp-probe-interval() seconds towards the primary server. If the primary logserver responds with a TCP ACK packet, the probe is successful. When the number of successful probes reaches the value set in the successful-probes-required() option, AxoSyslog tries to connect the primary server using the last probe.
Note
AxoSyslog always waits for the result of the last probe before sending the next message. So if one connection attempt takes longer than the configured interval, that is, it waits for connection time out, you may experience longer intervals between actual probes.
In the following example AxoSyslog attempts to return to the primary logserver, as set in the failback() option: it will check if the server is accessible every tcp-probe-interval() seconds, and reconnect to the primary logserver after three successful connection attempts.
Default value for tcp-probe-interval(): 60 seconds
Default value for successful-probes-required(): 3
Example: Configuring failover servers
In the following example AxoSyslog handles the logservers in round-robin fashion if the primary logserver becomes uneccassible (therefore failback() option is not set).
In the following example AxoSyslog attempts to return to the primary logserver, as set in the failback() option: it will check if the server is accessible every tcp-probe-interval() seconds, and reconnect to the primary logserver after three successful connection attempts.
This option is not available for the connection-less UDP protocol, because in this case the client does not detect that the destination becomes inaccessible.
flags()
Type:
no-multi-line, syslog-protocol
Default:
empty set
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
flush-lines()
Type:
number
Default:
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the <code>frac-digits()</code> option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).
Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).
For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.
Description: Specifies the Type-of-Service value of outgoing packets.
ip-ttl()
Type:
number
Default:
0
Description: Specifies the Time-To-Live value of outgoing packets.
keep-alive()
Type:
yes or no
Default:
yes
Description: Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.
localip()
Type:
string
Default:
0.0.0.0
Description: The IP address to bind to before connecting to target.
localport()
Type:
number
Default:
0
Description: The port number to bind to. Messages are sent from this port.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of <code>dst-idle</code>, <code>host-idle</code> and <code>periodical</code>, the <code>MARK</code> message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
port() or destport()
Type:
number
Default:
601
Description: The port number to connect to. Note that the default port numbers used by AxoSyslog do not comply with the latest RFC which was published after the release of AxoSyslog 3.0.2, therefore the default port numbers will change in the future releases.
so-broadcast()
Type:
yes or no
Default:
no
Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.
so-keepalive()
Type:
yes or no
Default:
no
Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-rcvbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.
so-sndbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.
spoof-source()
Type:
yes or no
Default:
no
Description: Enables source address spoofing. This means that the host running AxoSyslog generates UDP packets with the source IP address matching the original sender of the message. It is useful when you want to perform some kind of preprocessing using AxoSyslog then forward messages to your central log management solution with the source address of the original sender. This option only works for UDP destinations though the original message can be received by TCP as well. This option is only available if syslog-ng was compiled using the --enable-spoof-source configuration option.
The maximum size of spoofed datagrams in udp() destinations is set to 1024 bytes by default. To change the maximum size, use the spoof-source-max-msglen() option.
Note
<p>Anything above the size of the maximum transmission unit (MTU), which is 1500 bytes by default, is not recommended because of fragmentation.</p>
The maximum datagram in IP protocols (both IPv4 and IPv6) is 65535 bytes including the IP and UDP headers. The minimum size of the IPv4 header is 20 bytes, the IPv6 is 40 bytes, and the UDP is 8 bytes.
Warning
To use spoofing on Microsoft Windows platforms, you must also set the <code>spoof-interface()</code> option as well.
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
tcp-keepalive-intvl()
Type:
number [seconds]
Default:
0
Description: Specifies the interval (number of seconds) between subsequential keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.
Warning
<p>The <code>tcp-keepalive-time()</code>, <code>tcp-keepalive-probes()</code>, and <code>tcp-keepalive-intvl()</code> options only work on platforms which support the <code>TCP_KEEPCNT</code>, <code>TCP_KEEPIDLE</code>,and <code>TCP_KEEPINTVL</code> setsockopts. Currently, this is Linux.</p>
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
tcp-keepalive-probes()
Type:
number
Default:
0
Description: Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.
Warning
<p>The <code>tcp-keepalive-time()</code>, <code>tcp-keepalive-probes()</code>, and <code>tcp-keepalive-intvl()</code> options only work on platforms which support the <code>TCP_KEEPCNT</code>, <code>TCP_KEEPIDLE</code>,and <code>TCP_KEEPINTVL</code> setsockopts. Currently, this is Linux.</p>
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
tcp-keepalive-time()
Type:
number [seconds]
Default:
0
Description: Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.
Warning
<p>The <code>tcp-keepalive-time()</code>, <code>tcp-keepalive-probes()</code>, and <code>tcp-keepalive-intvl()</code> options only work on platforms which support the <code>TCP_KEEPCNT</code>, <code>TCP_KEEPIDLE</code>,and <code>TCP_KEEPINTVL</code> setsockopts. Currently, this is Linux.</p>
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
Note
If a message uses the IETF-syslog format (RFC5424), only the text of the message can be customized (that is, the $MESSAGE part of the log), the structure of the header is fixed.
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
transport()
Type:
udp, tcp, or tls
Default:
tcp
Description: Specifies the protocol used to send messages to the destination server.
If you use the udp transport, AxoSyslog automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, <code>network()</code>, or <code>syslog()</code>) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the <a href="/docs/axosyslog-core/chapter-global-options/reference-options/">proto-template</a> option.
8.43 - syslog-ng(): Forward logs to another syslog-ng node
The syslog-ng() destination driver forwards log messages to another AxoSyslog node in EWMM format.
The Enterprise-wide message model or EWMM allows you to deliver structured messages from the initial receiving AxoSyslog component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first AxoSyslog component in your infrastructure has received it, which is important if you want to forward a message for example, to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.
The syslog-ng() destination driver is available in version 3.16 and later. The node that receives this message must use the default-network-drivers() source to properly handle the messages.
The following is a sample log message in EWMM format.
<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
Note in this driver you have to set the address of the destination server using the server() parameter (in some other destinations, this parameter does not have an explicit name).
syslog-ng() destination options
The syslog-ng() destination is a special version of the network() destination driver: by default, it sends EWMM-formatted log messages to the TCP514 port of the server.
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Available only in AxoSyslog version 3.17 and later. For details about how client-side failover works, see Client-side failover.
servers()
Type:
list of IP addresses and fully-qualified domain names
Default:
empty
Description: Specifies a secondary destination server where log messages are sent if the primary server becomes inaccessible. To list several failover servers, separate the address of the servers with comma. By default, AxoSyslog waits for the a server before switching to the next failover server is set in the time-reopen() option.
If failback() is not set, AxoSyslog does not attempt to return to the primary server even if it becomes available. In case the failover server fails, AxoSyslog attempts to connect the next failover server in the list in round-robin fashion.
Warning
The failover servers must be accessible on the same port as the primary server.
failback()
Description: Available only in AxoSyslog version 3.17 and later.
When AxoSyslog starts up, it always connects to the primary server first. In the failover() option there is a possibility to customize the failover modes.
Depending on how you set the failback() option, AxoSyslog behaves as follows:
round-robin mode: If failback() is not set, AxoSyslog does not attempt to return to the primary server even if it becomes available. In case the failover server fails, AxoSyslog attempts to connect the next failover server in the list in round-robin fashion.
In the following example AxoSyslog handles the logservers in round-robin fashion if the primary logserver becomes inaccessible (therefore failback() option is not set).
failback mode: If failback() is set, AxoSyslog attempts to return to the primary server.
After AxoSyslog connects a secondary server during a failover, it sends a probe every tcp-probe-interval() seconds towards the primary server. If the primary logserver responds with a TCP ACK packet, the probe is successful. When the number of successful probes reaches the value set in the successful-probes-required() option, AxoSyslog tries to connect the primary server using the last probe.
Note
AxoSyslog always waits for the result of the last probe before sending the next message. So if one connection attempt takes longer than the configured interval, that is, it waits for connection time out, you may experience longer intervals between actual probes.
In the following example AxoSyslog attempts to return to the primary logserver, as set in the failback() option: it will check if the server is accessible every tcp-probe-interval() seconds, and reconnect to the primary logserver after three successful connection attempts.
Default value for tcp-probe-interval(): 60 seconds
Default value for successful-probes-required(): 3
Example: Configuring failover servers
In the following example AxoSyslog handles the logservers in round-robin fashion if the primary logserver becomes uneccassible (therefore failback() option is not set).
In the following example AxoSyslog attempts to return to the primary logserver, as set in the failback() option: it will check if the server is accessible every tcp-probe-interval() seconds, and reconnect to the primary logserver after three successful connection attempts.
This option is not available for the connection-less UDP protocol, because in this case the client does not detect that the destination becomes inaccessible.
Example: Specifying failover servers for syslog() destinations
The following example specifies two failover servers for a simple syslog() destination.
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
flush-lines()
Type:
number
Default:
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
ip-protocol()
Type:
number
Default:
4
Description: Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is ip-protocol(4).
Note that listening on a port using IPv6 automatically means that you are also listening on that port using IPv4. That is, if you want to have receive messages on an IP-address/port pair using both IPv4 and IPv6, create a source that uses the ip-protocol(6). You cannot have two sources with the same IP-address/port pair, but with different ip-protocol() settings (it causes an Address already in use error).
For example, the following source receives messages on TCP, using the network() driver, on every available interface of the host on both IPv4 and IPv6.
Description: Specifies the Type-of-Service value of outgoing packets.
ip-ttl()
Type:
number
Default:
0
Description: Specifies the Time-To-Live value of outgoing packets.
keep-alive()
Type:
yes or no
Default:
yes
Description: Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.
localip()
Type:
string
Default:
0.0.0.0
Description: The IP address to bind to before connecting to target.
localport()
Type:
number
Default:
0
Description: The port number to bind to. Messages are sent from this port.
log-fifo-size()
Type:
number
Default:
Use global setting.
Description: The number of messages that the output queue can store.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
port() or destport()
Type:
number
Default:
601
Description: The port number to connect to. Note that the default port numbers used by AxoSyslog do not comply with the latest RFC which was published after the release of AxoSyslog 3.0.2, therefore the default port numbers will change in the future releases.
server()
Type:
hostname or IP address
Default:
127.0.0.1
Description: The hostname or IP address of the AxoSyslog server.
so-broadcast()
Type:
yes or no
Default:
no
Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.
so-keepalive()
Type:
yes or no
Default:
no
Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-rcvbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.
so-sndbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
tcp-keepalive-intvl()
Type:
number [seconds]
Default:
0
Description: Specifies the interval (number of seconds) between subsequential keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.
Warning
The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
tcp-keepalive-probes()
Type:
number
Default:
0
Description: Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.
Warning
<p>The <code>tcp-keepalive-time()</code>, <code>tcp-keepalive-probes()</code>, and <code>tcp-keepalive-intvl()</code> options only work on platforms which support the <code>TCP_KEEPCNT</code>, <code>TCP_KEEPIDLE</code>,and <code>TCP_KEEPINTVL</code> setsockopts. Currently, this is Linux.</p>
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
tcp-keepalive-time()
Type:
number [seconds]
Default:
0
Description: Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.
Warning
<p>The <code>tcp-keepalive-time()</code>, <code>tcp-keepalive-probes()</code>, and <code>tcp-keepalive-intvl()</code> options only work on platforms which support the <code>TCP_KEEPCNT</code>, <code>TCP_KEEPIDLE</code>,and <code>TCP_KEEPINTVL</code> setsockopts. Currently, this is Linux.</p>
A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() \* tcp-keepalive-probes() seconds.
Available in AxoSyslog version 3.4 and later.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
tls()
Type:
tls options
Default:
n/a
Description: This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see TLS options.
transport()
Type:
udp, tcp, or tls
Default:
tcp
Description: Specifies the protocol used to send messages to the destination server.
If you use the udp transport, AxoSyslog automatically sends multicast packets if a multicast destination address is specified. The tcp transport does not support multicasting.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
8.44 - syslog-ng-otlp(): Forward logs to another node using OpenTelemetry
Available in AxoSyslog version 4.4 and later.
The syslog-ng-otlp() source and destination allows you to transfer the internal representation of log messages between AxoSyslog instances using the OpenTelemetry protocol. Unlike the traditional syslog-ng() drivers that rely on simple TCP connections, syslog-ng-otlp() leverages the OpenTelemetry protocol for efficient and reliable log message transmission.
The key benefits of the syslog-ng-otlp() drivers are:
scalability (via the workers() option),
built-in application layer acknowledgement,
support for Google service authentication (ADC or ALTS), and
improved load balancing capabilities.
To use it, configure a syslog-ng-otlp() destination on the sender node, and a syslog-ng-otlp() source on the receiver node, like this:
Application Layer Transport Security (ALTS) is a simple to use authentication, only available within Google’s infrastructure. It accepts the target-service-account() option, where you can list service accounts to match against when authenticating the server.
tls(peer-verify()) is not available for the opentelemetry() and loki() destination.
The gRPC-based drivers (opentelemetry() and loki()) have a different tls() block implementation from the network() or http() drivers. Most features are the same.
batch-bytes()
Accepted values:
number [bytes]
Default:
4MB
Available in AxoSyslog version 4.6 and later.
Description: Sets the maximum size of payload in a batch. If the size of the messages reaches this value, AxoSyslog sends the batch to the destination even if the number of messages is less than the value of the batch-lines() option. The batch might be at most 1 message larger than the set limit.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-bytes().
OTLP has a default 4 MiB batch limit, therefore the default value for batch-bytes() is 4 MB, which is a bit below 4 MiB.
The batch size is calculated before compression, which is the same as the limit is calculated on the server.
batch-lines()
Type:
number
Default:
0
Description: Specifies how many lines are flushed to a destination in one batch. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
For example, if you set batch-lines() to 100, AxoSyslog waits for 100 messages.
If the batch-timeout() option is disabled, the AxoSyslog application flushes the messages if it has sent batch-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
Note that if the batch-timeout() option is enabled and the queue becomes empty, AxoSyslog flushes the messages only if batch-timeout() expires, or the batch reaches the limit set in batch-lines().
For optimal performance, make sure that the AxoSyslog source that feeds messages to this destination is configured properly: the value of the log-iw-size() option of the source must be higher than the batch-lines()*workers() of the destination. Otherwise, the size of the batches cannot reach the batch-lines() limit.
batch-timeout()
Type:
time in milliseconds
Default:
-1 (disabled)
Description: Specifies the time AxoSyslog waits for lines to accumulate in the output buffer. The AxoSyslog application sends batches to the destinations evenly. The timer starts when the first message arrives to the buffer, so if only few messages arrive, AxoSyslog sends messages to the destination at most once every batch-timeout() milliseconds.
channel-args()
Type:
See description
Default:
-
Description: The channel-args() option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the GRPC Core library documentation. For example:
Description: Enables compression in gRPC requests. Although gRPC supports various compression methods, currently only deflate is supported (which is basically the same as gzip).
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The number of messages that the output queue can store.
persist-name()
Type:
string
Default:
N/A
Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").
retries()
Type:
number (of attempts)
Default:
3
Description: If AxoSyslog cannot send a message, it will try again until the number of attempts reaches retries().
If the number of attempts reaches retries(), AxoSyslog will wait for time-reopen() time, then tries sending the message again.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
url()
Type:
string
Default:
localhost:9095
Description: The URL of the AxoSyslog receiver.
worker-partition-key()
Type:
template
Default:
Description: The worker-partition-key() option specifies a template: messages that expand the template to the same value are mapped to the same partition. When batching is enabled and multiple workers are configured, it’s important to add only those messages to a batch which generate identical URLs. To achieve this, set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed.
For example, you can partition messages based on the destination host:
worker-partition-key("$HOST")
workers()
Type:
integer
Default:
1
Description: Specifies the number of worker threads (at least 1) that AxoSyslog uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
Warning
Hazard of data loss. When you use more than one worker threads together with disk-based buffering, AxoSyslog creates a separate disk buffer for each worker thread. This means that decreasing the number of workers can result in losing data currently stored in the disk buffer files. Do not decrease the number of workers when the disk buffer files are in use.
8.45 - tcp, tcp6, udp, udp6: OBSOLETE - Send messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)
The tcp(), tcp6(), udp(), and udp6() drivers send messages to another host (for example, an AxoSyslog server or relay) on the local intranet or internet using the UDP or TCP protocol. The tcp6() and udp6() drivers use the IPv6 network protocol.
8.45.1 - tcp(), tcp6(), udp(), and udp6() destination options
8.45.1.1 - Change an old destination driver to the network() driver
To replace your existing tcp(), tcp6(), udp(), udp6() destinations with a network() destination, complete the following steps.
Replace the driver with network. For example, replace udp( with network(
Set the transport protocol.
If you used TLS-encryption, add the transport("tls") option, then continue with the next step.
If you used the tcp or tcp6 driver, add the transport("tcp") option.
If you used the udp or udp driver, add the transport("udp") option.
If you use IPv6 (that is, the udp6 or tcp6 driver), add the ip-protocol(6) option.
If you did not specify the port used in the old driver, check network() destination options and verify that your clients send the messages to the default port of the transport protocol you use. Otherwise, set the appropriate port number in your source using the port() option.
All other options are identical. Test your configuration with the syslog-ng --syntax-only command.
The following configuration shows a simple tcp destination.
The telegram() destination sends log messages to Telegram, which is a secure, cloud-based mobile and desktop messaging app.
Note that this destination automatically uses the certificate store of the system (for details, see the curl documentation).
Declaration:
telegram(parameters);
You can use the proxy() option to configure the HTTP driver in all HTTP-based destinations to use a specific HTTP proxy that is independent from the proxy configured for the system.
Example: Using the telegram() driver
The following example creates a telegram() destination.
Description: Disables link previews for links in the message. By default, the disable-web-page-preview value is true. From a security point of view, Axoflow recommends to leave it true, otherwise malicious messages can trick the telegram destination to generate traffic to any URL.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: Formats the message in a markdown-style or HTML-style formatting. By default, the parse-mode value is markdown, which means that the message is formatted in markdown style.
template()
Type:
string
Default:
${MESSAGE}
Description: Specifies the content of the message. The AxoSyslog application will automatically encode the content of this option using the url-encode() template function.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
8.47 - unix-stream, unix-dgram: Send messages to UNIX domain sockets
The unix-stream() and unix-dgram() drivers send messages to a UNIX domain socket in either SOCK_STREAM or SOCK_DGRAM mode.
8.47.1 - unix-stream() and unix-dgram() destination options
These drivers send messages to a unix socket in either SOCK_STREAM or SOCK_DGRAM mode. The unix-stream() and unix-dgram() destinations have the following options:
close-on-input()
Type:
yes
Default:
yes
Description: By default, AxoSyslog closes destination sockets if it receives any input from the socket (for example, a reply). If this option is set to no, AxoSyslog just ignores the input, but does not close the socket.
create-dirs()
Type:
yes or no
Default:
no
Description: Enable creating non-existing directories when creating files or socket files.
disk-buffer()
Description: This option enables putting outgoing messages into the disk buffer of the destination to avoid message loss in case of a system failure on the destination side. It has the following options:
capacity-bytes()
Type:
number (bytes)
Default:
1MiB
Description: This is a required option. The maximum size of the disk-buffer in bytes. The minimum value is 1048576 bytes. If you set a smaller value, the minimum value will be used automatically. It replaces the old log-disk-fifo-size() option.
In AxoSyslog version 4.2 and earlier, this option was called disk-buf-size().
compaction()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog prunes the unused space in the LogMessage representation, making the disk queue size smaller at the cost of some CPU time. Setting the compaction() argument to yes is recommended when numerous name-value pairs are unset during processing, or when the same names are set multiple times.
Note
Simply unsetting these name-value pairs by using the unset() rewrite operation is not enough, as due to performance reasons that help when AxoSyslog is CPU bound, the internal representation of a LogMessage will not release the memory associated with these name-value pairs. In some cases, however, the size of this overhead becomes significant (the raw message size can grow up to four times its original size), which unnecessarily increases the disk queue file size. For these cases, the compaction will drop unset values, making the LogMessage representation smaller at the cost of some CPU time required to perform compaction.
dir()
Type:
string
Default:
N/A
Description: Defines the folder where the disk-buffer files are stored.
Warning
When creating a new dir() option for a disk buffer, or modifying an existing one, make sure you delete the persist file.
AxoSyslog creates disk-buffer files based on the path recorded in the persist file. Therefore, if the persist file is not deleted after modifying the dir() option, then following a restart, AxoSyslog will look for or create disk-buffer files in their old location. To ensure that AxoSyslog uses the new dir() setting, the persist file must not contain any information about the destinations which the disk-buffer file in question belongs to.
Note
If the dir() path provided by the user does not exist, AxoSyslog creates the path with the same permission as the running instance.
flow-control-window-bytes()
Type:
number (bytes)
Default:
163840000
Description: Use this option if the option reliable() is set to yes. This option contains the size of the messages in bytes that is used in the memory part of the disk buffer. It replaces the old log-fifo-size() option. It does not inherit the value of the global log-fifo-size() option, even if it is provided. Note that this option will be ignored if the option reliable() is set to no.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-size().
flow-control-window-size()
Type:
number(messages)
Default:
10000
Description: Use this option if the option reliable() is set to no. This option contains the number of messages stored in overflow queue. It replaces the old log-fifo-size() option. It inherits the value of the global log-fifo-size() option if provided. If it is not provided, the default value is 10000 messages. Note that this option will be ignored if the option reliable() is set to yes.
In AxoSyslog version 4.2 and earlier, this option was called mem-buf-length().
front-cache-size()
Type:
number(messages)
Default:
1000
Description: The number of messages stored in the output buffer of the destination. Note that if you change the value of this option and the disk-buffer already exists, the change will take effect when the disk-buffer becomes empty.
Options reliable() and capacity-bytes() are required options.
In AxoSyslog version 4.2 and earlier, this option was called qout-size().
prealloc()
Type:
yes/no
Default:
no
Description:
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
Available in AxoSyslog 4.0 and later.
reliable()
Type:
yes/no
Default:
no
Description: If set to yes, AxoSyslog cannot lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.
Warning
Hazard of data loss! If you change the value of reliable() option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost.
truncate-size-ratio()
Type:
number((between 0 and 1))
Default:
1 (do not truncate)
Description: Limits the truncation of the disk-buffer file. Truncating the disk-buffer file can slow down the disk IO operations, but it saves disk space. By default, AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default (truncate-size-ratio(1)). Earlier versions freed the disk-space when at least 10% of the disk-buffer file could be freed (truncate-size-ratio(0.1)).
AxoSyslog only truncates the file if the possible disk gain is more than truncate-size-ratio() times capacity-bytes().
Description: Flags influence the behavior of the destination driver.
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line.
syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.
flush-lines()
Type:
number
Default:
Use global setting (exception: for http() destination, the default is 1).
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an AxoSyslog server, make sure that the value of flush-lines() is smaller than the window size set in the log-iw-size() option in the source of your server.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
hook-commands()
Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands() can be used with all source and destination drivers with the exception of the usertty() and internal() drivers.
Note
The AxoSyslog application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor or SELinux, you might have to modify your AppArmor or SELinux configuration to enable AxoSyslog to execute external applications.
Using hook-commands() when AxoSyslog starts or stops
To execute an external program when AxoSyslog starts or stops, use the following options:
startup()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog starts.
shutdown()
Type:
string
Default:
N/A
Description: Defines the external program that is executed as AxoSyslog stops.
Using the hook-commands() when AxoSyslog reloads
To execute an external program when the AxoSyslog configuration is initiated or torn down, for example, on startup/shutdown or during a AxoSyslog reload, use the following options:
setup()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is initiated, for example, on startup or during a AxoSyslog reload.
teardown()
Type:
string
Default:
N/A
Description: Defines an external program that is executed when the AxoSyslog configuration is stopped or torn down, for example, on shutdown or during a AxoSyslog reload.
Example: Using the hook-commands() with a network source
In the following example, the hook-commands() is used with the network() driver and it opens an iptables port automatically as AxoSyslog is started/stopped.
The assumption in this example is that the LOGCHAIN chain is part of a larger ruleset that routes traffic to it. Whenever the AxoSyslog created rule is there, packets can flow, otherwise the port is closed.
Description: The number of messages that the output queue can store.
keep-alive()
Type:
yes or no
Default:
yes
Description: Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
so-broadcast()
Type:
yes or no
Default:
no
Description: This option controls the SO_BROADCAST socket option required to make AxoSyslog send messages to a broadcast address. For details, see the socket(7) manual page.
so-keepalive()
Type:
yes or no
Default:
no
Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.
so-rcvbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.
so-sndbuf()
Type:
number
Default:
0
Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.
suppress()
Type:
seconds
Default:
0 (disabled)
Description: If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), AxoSyslog can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds AxoSyslog waits for identical messages.
template()
Type:
string
Default:
A format conforming to the default logfile format.
Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of AxoSyslog. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template-escape()
Type:
yes or no
Default:
no
Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
throttle()
Type:
number
Default:
0
Description: Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
ts-format()
Type:
rfc3164, bsd, rfc3339, iso
Default:
rfc3164
Description: Override the global timestamp format (set in the global ts-format() parameter) for the specific destination. For details, see ts-format().
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
8.48 - usertty: Send messages to a user terminal
This driver writes messages to the terminal of a logged-in user.
The usertty() driver has a single required argument, specifying a username who should receive a copy of matching messages. Use the asterisk * to specify every user currently logged in to the system.
Declaration:
usertty(username);
The usertty() does not have any further options nor does it support templates.
Example: Using the usertty() driver
destination d_usertty { usertty("root");};
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
8.49 - Write your own custom destination in Java or Python
The AxoSyslog application is open source, so if you have the necessary programming skills, you can extend it if its features are not adequate for your particular environment or needs. You can write destinations and other extensions to AxoSyslog in C (the main language of AxoSyslog), or using its language bindings, for example, Java or Python. .
If you delete all Java destinations from your configuration and reload syslog-ng, the JVM is not used anymore, but it is still running. If you want to stop JVM, stop syslog-ng and then start syslog-ng again.
8.50 - Client-side failover
AxoSyslog can detect if the remote server of a network destination becomes inaccessible, and start sending messages to a secondary server. You can configure multiple failover servers, so if the secondary server becomes inaccessible as well, AxoSyslog switches to the third server in the list, and so on. If there are no more failover servers left, AxoSyslog returns to the beginning of a list and attempts to connect to the primary server.
The primary server is the address you provided in the destination driver configuration and it has a special role. AxoSyslog nominates this destination over the failover servers, and handles it as the primary address.
When AxoSyslog starts up, it always connects to the primary server first. In the failover() option there is a possibility to customize the failover modes.
Depending on how you set the failback() option, AxoSyslog behaves as follows:
round-robin mode: If failback() is not set, AxoSyslog does not attempt to return to the primary server even if it becomes available. In case the failover server fails, AxoSyslog attempts to connect the next failover server in the list in round-robin fashion.
In the following example AxoSyslog handles the logservers in round-robin fashion if the primary logserver becomes inaccessible (therefore failback() option is not set).
failback mode: If failback() is set, AxoSyslog attempts to return to the primary server.
After AxoSyslog connects a secondary server during a failover, it sends a probe every tcp-probe-interval() seconds towards the primary server. If the primary logserver responds with a TCP ACK packet, the probe is successful. When the number of successful probes reaches the value set in the successful-probes-required() option, AxoSyslog tries to connect the primary server using the last probe.
Note
AxoSyslog always waits for the result of the last probe before sending the next message. So if one connection attempt takes longer than the configured interval, that is, it waits for connection time out, you may experience longer intervals between actual probes.
In the following example AxoSyslog attempts to return to the primary logserver, as set in the failback() option: it will check if the server is accessible every tcp-probe-interval() seconds, and reconnect to the primary logserver after three successful connection attempts.
If AxoSyslog is restarted, it attempts to connect the primary server.
If AxoSyslog uses TLS-encryption to communicate with the remote server, AxoSyslog checks the certificate of the failover server as well. The certificates of the failover servers should match their domain names or IP addresses — for details, see Encrypting log messages with TLS. Note that when mutual authentication is used, the AxoSyslog client sends the same certificate to every server.
The primary server and the failover servers must be accessible with the same communication method: it is not possible to use different destination drivers or options for the different servers.
Note
Client-side failover works only for TCP-based connections (including TLS-encrypted connections), that is, the syslog() and network() destination drivers (excluding UDP transport).
Client-side failover is not supported in the sql() driver, even though it may use a TCP connection to access a remote database.
9 - log: Filter and route log messages using log paths, flags, and filters
9.1 - Log paths
Log paths determine what happens with the incoming log messages. Messages coming from the sources listed in the log statement and matching all the filters are sent to the listed destinations.
To define a log path, add a log statement to the syslog-ng configuration file using the following syntax:
Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags.
Note
The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.
Named log paths and log path metrics
In AxoSyslog version 4.1 and later, you can add an ID or name to the log path to make the configuration file more readable. Also, AxoSyslog collects ingress and egress metrics for named log paths. For example:
All matching log statements are processed by default, and the messages are sent to every matching destination by default. So a single log message might be sent to the same destination several times, provided the destination is listed in several log statements, and it can be also sent to several different destinations.
This default behavior can be changed using the flags() parameter. Flags apply to individual log paths, they are not global options. For details and examples on the available flags, see Log path flags. The effect and use of the flow-control flag is detailed in Managing incoming and outgoing messages with flow-control.
9.1.1 - Embedded log statements
Starting from version 3.0, AxoSyslog can handle embedded log statements (also called log pipes). Embedded log statements are useful for creating complex, multi-level log paths with several destinations and use filters, parsers, and rewrite rules.
For example, if you want to filter your incoming messages based on the facility parameter, and then use further filters to send messages arriving from different hosts to different destinations, you would use embedded log statements.
Embedded log statements include sources — and usually filters, parsers, rewrite rules, or destinations — and other log statements that can include filters, parsers, rewrite rules, and destinations. The following rules apply to embedded log statements:
Only the beginning (also called top-level) log statement can include sources.
Embedded log statements can include multiple log statements on the same level (that is, a top-level log statement can include two or more log statements).
Embedded log statements can include several levels of log statements (that is, a top-level log statement can include a log statement that includes another log statement, and so on).
After an embedded log statement, you can write either another log statement, or the flags() option of the original log statement. You cannot use filters or other configuration objects. This also means that flags (except for the flow-control flag) apply to the entire log statement, you cannot use them only for the embedded log statement.
Embedded log statements that are on the same level receive the same messages from the higher-level log statement. For example, if the top-level log statement includes a filter, the lower-level log statements receive only the messages that pass the filter.
Embedded log filters can be used to optimize the processing of log messages, for example, to re-use the results of filtering and rewriting operations.
9.1.1.1 - Using embedded log statements
Embedded log statements (for details, see Embedded log statements) re-use the results of processing messages (for example, the results of filtering or rewriting) to create complex log paths. Embedded log statements use the same syntax as regular log statements, but they cannot contain additional sources. To define embedded log statements, use the following syntax:
The following log path sends every message to the configured destinations: both the d_file1 and the d_file2 destinations receive every message of the source.
The following example collects logs from multiple source groups and uses the source() filter in the embedded log statement to select messages of the s_network source group.
This format considers all filters and all parsers as the condition, combined. If the message contains 'foo' and the date-parser() fails, the else branch is taken. Similarly, if the message does not contain 'foo', the else branch is taken.
Using the if {} and else {} blocks in your configuration
You can copy-paste the following example and use it as a template for using the if {} and else {} blocks in your configuration.
Example for using the if {} and else {} blocks in your configuration
The following configuration can be used as a template for using the if {} and else {} blocks:
An alternative, less straightforward way to implement conditional evaluation is to use junctions. For details on junctions and channels, see Junctions and channels.
9.1.3 - Junctions and channels
Junctions make it possible to send the messages to different channels, process the messages differently on each channel, and then join every channel together again. You can define any number of channels in a junction: every channel receives a copy of every message that reaches the junction. Every channel can process the messages differently, and at the end of the junction, the processed messages of every channel return to the junction again, where further processing is possible.
A junction includes one or more channels. A channel usually includes at least one filter, though that is not enforced. Otherwise, channels are identical to log statements, and can include any kind of objects, for example, parsers, rewrite rules, destinations, and so on. (For details on using channels, as well as on using channels outside junctions, see Using channels in configuration objects.)
Note
Certain parsers can also act as filters:
The JSON parser automatically discards messages that are not valid JSON messages.
The csv-parser() discards invalid messages if the flags(drop-invalid) option is set.
You can also use log-path flags in the channels of the junction. Within the junction, a message is processed by every channel, in the order the channels appear in the configuration file. Typically if your channels have filters, you also set the flags(final) option for the channel. However, note that the log-path flags of the channel apply only within the junction, for example, if you set the final flag for a channel, then the subsequent channels of the junction will not receive the message, but this does not affect any other log path or junction of the configuration. The only exception is the flow-control flag: if you enable flow-control in a junction, it affects the entire log path. For details on log-path flags, see Log path flags.
For example, suppose that you have a single network source that receives log messages from different devices, and some devices send messages that are not RFC-compliant (some routers are notorious for that). To solve this problem in earlier versions of AxoSyslog, you had to create two different network sources using different IP addresses or ports: one that received the RFC-compliant messages, and one that received the improperly formatted messages (for example, using the flags(no-parse) option). Using junctions this becomes much more simple: you can use a single network source to receive every message, then use a junction and two channels. The first channel processes the RFC-compliant messages, the second everything else. At the end, every message is stored in a single file. The filters used in the example can be host() filters (if you have a list of the IP addresses of the devices sending non-compliant messages), but that depends on your environment.
Since every channel receives every message that reaches the junction, use the flags(final) option in the channels to avoid the unnecessary processing the messages multiple times:
Junctions differ from embedded log statements, because embedded log statements are like branches: they split the flow of messages into separate paths, and the different paths do not meet again. Messages processed on different embedded log statements cannot be combined together for further processing. However, junctions split the messages to channels, then combine the channels together.
An alternative, more straightforward way to implement conditional evaluation is to configure conditional expressions using if {}, elif {}, and else {} blocks. For details, see if-else-elif: Conditional expressions.
9.1.4 - Log path flags
Flags influence the behavior of syslog-ng, and the way it processes messages. The following flags may be used in the log paths, as described in Log paths.
Log statement flags
catchall
This flag means that the source of the message is ignored, only the filters of the log path are taken into account when matching messages. A log statement using the catchall flag processes every message that arrives to any of the defined sources.
drop-unmatched
This flag means that the message is dropped along a log path when it does not match a filter or is discarded by a parser. Without using the drop-unmatched flag, AxoSyslog would continue to process the message along alternative paths.
fallback
This flag makes a log statement ‘fallback’. Fallback log statements process messages that were not processed by other, ’non-fallback’ log statements.
Processed means that every filter of a log path matched the message. Note that in the case of embedded log paths, the message is considered to be processed if it matches the filters of the outer log path, even if it does not match the filters of the embedded log path. For details, see Example: Using log path flags.
final
This flag means that the processing of log messages processed by the log statement ends here, other log statements appearing later in the configuration file will not process the messages processed by the log statement labeled as ‘final’. Note that this does not necessarily mean that matching messages will be stored only once, as there can be matching log statements processed before the current one (AxoSyslog evaluates log statements in the order they appear in the configuration file).
Processed means that every filter of a log path matched the message. Note that in the case of embedded log paths, the message is considered to be processed if it matches the filters of the outer log path, even if it does not match the filters of the embedded log path. For details, see Example: Using log path flags.
flow-control
Enables flow-control to the log path, meaning that AxoSyslog will stop reading messages from the sources of this log statement if the destinations are not able to process the messages at the required speed. If disabled, AxoSyslog will drop messages if the destination queues are full. If enabled, AxoSyslog will only drop messages if the destination queues/window sizes are improperly sized. For details, see Managing incoming and outgoing messages with flow-control.
Warning
The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.
Example: Using log path flags
Let’s suppose that you have two hosts (myhost_A and myhost_B) that run two applications each (application_A and application_B), and you collect the log messages to a central AxoSyslog server. On the server, you create two log paths:
one that processes only the messages sent by myhost_A, and
one that processes only the messages sent by application_A.
This means that messages sent by application_A running on myhost_A will be processed by both log paths, and the messages of application_B running on myhost_B will not be processed at all.
If you add the final flag to the first log path, then only this log path will process the messages of myhost_A, so the second log path will receive only the messages of application_A running on myhost_B.
If you create a third log path that includes the fallback flag, it will process the messages not processed by the first two log paths, in this case, the messages of application_B running on myhost_B.
Adding a fourth log path with the catchall flag would process every message received by the AxoSyslog server.
The following example shows a scenario that can result in message loss. Do NOT use such a configuration, unless you know exactly what you are doing. The problem is if a message matches the filters in the first part of the first log path, AxoSyslog treats the message as ‘processed’. Since the first log path includes the final flag, AxoSyslog will not pass the message to the second log path (the one with the fallback flag). As a result, AxoSyslog drops messages that do not match the filter of the embedded log path.
# Do not use such a configuration, unless you know exactly what you are doing. log { source(s_network);# Filters in the external log path.# If a message matches this filter, it is treated as 'processed' filter(f_program); filter(f_message); log {# Filter in the embedded log path.# If a message does not match this filter, it is lost, it will not be processed by the 'fallback' log path filter(f_host); destination(d_file1);}; flags(final);}; log { source(s_network); destination(d_file2); flags(fallback);};
Example: Using the drop-unmatched flag
In the following example, if a log message arrives whose $MSG part does not contain the string foo, then AxoSyslog will discard the message and will not check compliance with the second if condition.
(Without the drop-unmatched flag, AxoSyslog would check if the message complies with the second if condition, that is, whether or not the message contains the string bar .)
9.2 - Managing incoming and outgoing messages with flow-control
This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message losses.
The AxoSyslog application monitors (polls) the sources defined in its configuration file, periodically checking each source for messages. When a log message is found in one of the sources, AxoSyslog polls every source and reads the available messages. These messages are processed and put into the output buffer of AxoSyslog (also called fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.
In large-traffic environments many messages can arrive during a single poll loop, therefore AxoSyslog reads only a fixed number of messages from each source. The log-fetch-limit() option specifies the number of messages read during a poll loop from a single source.
TCP and unix-stream sources can receive the logs from several incoming connections (for example, many different clients or applications). For such sources, AxoSyslog reads messages from every connection, thus the log-fetch-limit() parameter applies individually to every connection of the source.
Log paths without flow-control
Every destination has its own output buffer. The output buffer is needed because the destination might not be able to accept all messages immediately. The log-fifo-size() parameter sets the size of the output buffer. The output buffer must be larger than the log-fetch-limit() of the sources, to ensure that every message read during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output buffer must be large enough to store the incoming messages of every source.
Log paths with flow-control
The AxoSyslog application uses flow-control in the following cases:
Hard flow-control: the flow-control flag is enabled for the particular log path.
Soft flow-control: the log path includes a file destination.
Note
The way flow-control works has changed significantly in version AxoSyslog 3.22. If you are using an older version of AxoSyslog, consult the documentation of the version you are using for details about flow-control.
The flow-control of AxoSyslog introduces a control window to the source that tracks how many messages can AxoSyslog accept from the source. Every message that AxoSyslog reads from the source lowers the window size by one, every message that AxoSyslog successfully sends from the output buffer increases the window size by one. If the window is full (that is, its size decreases to zero), AxoSyslog stops reading messages from the source. The initial size of the control window is by default 100. If a source accepts messages from multiple connections, all messages use the same control window.
When using flow-control, AxoSyslog automatically sets the size of the output buffer so that it matches the size of the control window of the sources. Note that starting with AxoSyslog 3.22, log-fifo-size() only affects log paths that are not flow-controlled.
Note
If the source can handle multiple connections (for example, network() and syslog()), the size of the control window is divided by the value of the max-connections() parameter and this smaller control window is applied to each connection of the source.
Dynamic flow-control
In addition to the static control window set using the log-iw-size() option, you can also allocate a dynamic window to the source. The AxoSyslog application uses this window to dynamically increase the static window of the active connections. The dynamic window is distributed evenly among the active connections of the source. The AxoSyslog application periodically checks which connections of the source are active, and redistributes the dynamic window. If only one of the connections is active, it receives the entire dynamic window, while other connections receive only their share of the static window.
Using dynamic flow-control on your AxoSyslog server is useful when the source has lots of connections, but only a small subset of the active clients send messages at high rate, and the memory of the AxoSyslog server is limited. In other cases, it is currently not recommended, because it can result in higher memory usage and fluctuating performance compared to using only the static window.
When flow-control is used, every source has its own control window. As a worst-case situation, memory of the host must be greater than the total size of the messages of every control window, plus the size of the dynamic window, that is, the log-iw-size()+dynamic-window-size(). This applies to every source that sends logs to the particular destination. Thus if two sources having several connections and heavy traffic send logs to the same destination, the control window of both sources must fit into the memory of the host. Otherwise, some messages might not fit in the memory, and messages may be lost.
If dynamic flow-control is disabled (which is the default behavior), the value of the log-iw-size() option cannot be lower than 100. If dynamic flow-control is enabled, you can decrease the value of the log-iw-size() option (to the minimum of 1).
In case of soft flow-control there is no message lost if the destination can accept messages. It is possible to lose messages if it cannot accept messages (for example, the file destination is not writable, or the disk becomes full), and all buffers are full. Soft flow-control cannot be configured, it is automatically available for file destinations.
Hard flow-control: In case of hard flow-control there is no message lost. To use hard flow-control, enable the flow-control flag in the log path. Hard flow-control is available for all destinations.
Hazard of data loss! For destinations other than file, soft flow-control is not available. Thus, it is possible to lose log messages on those destinations. To avoid data loss on those destinations, use hard flow-control.
The AxoSyslog application handles outgoing messages the following way:
Output queue: Messages from the output queue are sent to the target AxoSyslog server. The AxoSyslog application puts the outgoing messages directly into the output queue, unless the output queue is full. The output queue can hold 64 messages, this is a fixed value and cannot be modified.
Disk buffer: If the output queue is full and disk-buffering is enabled, AxoSyslog puts the outgoing messages into the disk buffer of the destination.
Overflow queue: If the output queue is full and the disk buffer is disabled or full, AxoSyslog puts the outgoing messages into the overflow queue of the destination. (The overflow queue is identical to the output buffer used by other destinations.) The log-fifo-size() parameter specifies the number of messages stored in the overflow queue, unless flow-control is enabled. When dynamic flow-control is enabled, AxoSyslog sets the size of the overflow queue automatically. For details on sizing the log-fifo-size() parameter, see Configuring flow-control.
9.2.1 - Flow-control and multiple destinations
Using flow-control on a source has an important side-effect if the messages of the source are sent to multiple destinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because AxoSyslog stops reading the source. For example, if messages from a source are sent to a remote server and also stored locally in a file, and the network connection to the server becomes unavailable, neither the remote server nor the local file will receive any messages.
Note
Creating separate log paths for the destinations that use the same flow-controlled source does not avoid the problem.
If you use flow-control and reliable disk-based buffering together with multiple destinations, the flow-control starts slowing down the source only when:
one destination is down, and
the number of messages stored in the disk buffer of the destination reaches (capacity-bytes() minus flow-control-window-bytes()).
The AxoSyslog application normally reads a maximum of log-fetch-limit() number of messages from a source.
From TCP and unix-stream sources, AxoSyslog reads a maximum of log-fetch-limit() from every connection of the source. The number of connections to the source is set using the max-connections() parameter.
Every destination has an output buffer. The size of this buffer is set automatically for log paths that use flow-control, and can be set using the log-fifo-size() option for other log paths.
Flow-control uses a control window to determine if there is free space in the output buffer for new messages. Every source has its own control window, the log-iw-size() option sets the size of the static control window. Optionally, you can enable a dynamic control window for the source using the dynamic-window-size() option.
When a source accepts multiple connections, the size of the control window is divided by the value of the max-connections() parameter and this smaller control window is applied to each connection of the source.
The dynamic control window is automatically distributed among the active connections of the source.
If the control window is full, AxoSyslog stops reading messages from the source until some messages are successfully sent to the destination.
If the output buffer becomes full, and neither disk-buffering nor flow-control is used, messages may be lost.
Warning
If you modify the max-connections() or the log-fetch-limit() parameter, do not forget to adjust the log-iw-size() and dynamic-window-size() parameters accordingly.
Example: Sizing parameters for flow-control
Suppose that AxoSyslog has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket.
Set the max-connections() parameter of the source to 300. However, the log-fetch-limit() (default value: 10) parameter applies to every connection of the source individually, while the log-iw-size() (default value: 1000) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log-fetch-limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max-connections()*log-fetch-limit() messages to be able to read every incoming message of a poll loop. In the current example this means that log-iw-size() should be greater than 300\*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing AxoSyslog to read only one message of the last connections in every poll loop.
The output buffer of the destination must accommodate at least log-iw-size() messages, but use a greater value: in the current example 3000\*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, AxoSyslog does not read any messages from the source until some messages are successfully sent to the destination.
If other sources send messages to this destination, then the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, then increase the log-fifo-size() by 10000.
The AxoSyslog application can store messages on the local hard disk if the destination (for example, the central log server) or the network connection to the destination becomes unavailable. The AxoSyslog application automatically sends the stored messages to the destination when the connection is reestablished. The disk buffer is used as a queue: when the connection to the destination is reestablished, AxoSyslog sends the messages to the destination in the order they were received.
Every destination driver supports the disk-buffer() option. The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method.
Every such destination uses a separate disk buffer (similarly to the output buffers controlled by log-fifo-size()). The hard disk space is not pre-allocated, so ensure that there is always enough free space to store the disk buffers even when the disk buffers are full.
If AxoSyslog is restarted (using the /etc/init.d/syslog-ng restart command, or another appropriate command on your platform), it automatically saves any unsent messages from the disk buffer and in-memory queues. After the restart, AxoSyslog sends the saved messages to the destination. In other words, the disk buffer is persistent. The disk buffer is also resistant to AxoSyslog crashes.
When you use disk-based buffering, and the reliable() option is set to no, AxoSyslog handles outgoing messages the following way:
Output queue: In-memory queue. If there is space left in it, AxoSyslog puts the message into this queue first . Messages stored here are processed faster, because AxoSyslog can skip writing to, and reading from the disk, as well as serializing or deserializing the message, saving I/O and processor time as a result. The contents of the in-memory output queue are persisted to the disk-buffer file during AxoSyslog reload, restart or stop, but they cannot be persisted if in the event of power failures, or if AxoSyslog crashes. By default, the output queue can hold 1000 messages (you can adjust this number using the quot-size() option).
Disk-buffer file: Disk queue. If there is no space left in the output queue, the message is stored on the disk-buffer file. Messages stored here are persisted on the disk, even in case of power failures or if AxoSyslog crashes. Using the disk-buffer file takes considerable amount of disk I/O and processor time. The size of this queue can be set with the capacity-bytes() option.
Overflow queue: In-memory queue. This queue is used to trigger flow-control if it is set. The contents of the in-memory overflow queue are persisted to the disk-buffer file in case of AxoSyslog reload, restart or stop, but they are not persisted in case of power failures or if AxoSyslog crashes. Setting the size of the overflow queue can be done with the flow-control-window-size() option.
Warning
Hazard of data loss!
In case of normal disk-buffers, the messages stored in the output queue and the overflow queue can be lost in case of power failures or if AxoSyslog crashes.
Note
Using disk buffer can significantly decrease performance.
Message handling and reliable disk-based buffering
When you use disk-based buffering, and the reliable() option is set to yes, AxoSyslog handles outgoing messages the following way.
The flow-control-window-bytes() option determines when flow-control is triggered. After the size of the disk-buffer file reaches (capacity-bytes() minus flow-control-window-bytes()), messages are written into both the disk-buffer file and the overflow queue, indicating that flow-control needs to slow down the message source. These messages are not taken out from the control window (governed by log-iw-size()), causing the control window to fill up.
If the control window is full, the flow-control completely stops reading incoming messages from the source. (As a result, flow-control-window-bytes() must be at least as large as log-iw-size() times the average message size.)
Output queue: In-memory and disk queue. If there is space left in it, AxoSyslog puts the message into this queue first. In case of reliable disk-buffer, in addition to storing the message in memory, it is stored directly in the disk-buffer file as well for safety reasons (see the next point). Messages stored here are processed faster, because AxoSyslog can skip reading from the disk, and deserializing the message, saving I/O and processor time. By default, the output queue can hold 1000 messages (you can adjust it using the quot-size() option).
Disk-buffer file: Disk queue. If there is no space left in the output queue, the message is stored on the disk-buffer file. Messages stored here are persisted on the disk, and survive AxoSyslog crash or power failure. Using the disk-buffer file takes considerable amount of disk I/O and processor time. The size of this queue can be set with the capacity-bytes() option.
Overflow queue: In-memory and disk queue. This queue is used to trigger flow-control if it is set. Similarly to the output queue, in case of reliable disk-buffer in addition to storing the message in memory, it is stored directly in the disk-buffer file as well for safety reasons. Setting the size of the overflow queue can be done with the flow-control-window-bytes() option.
9.3.1 - Enabling reliable disk-based buffering
Every destination driver supports the disk-buffer() option. The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method.
To enable reliable disk-based buffering, use the disk-buffer(reliable(yes)) parameter in the destination. Use reliable disk-based buffering if you do not want to lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. The filename of the reliable disk buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.rqf.
Example: Example for using reliable disk-based buffering
For details on the differences between normal and reliable disk-based buffering, see also About disk queue files.
9.3.2 - Enabling normal disk-based buffering
Every destination driver supports the disk-buffer() option. The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method.
If the reliable() option is not set, by default a normal disk-buffer is created. To explicitly enable the normal disk-buffer option, use the disk-buffer(reliable(no)) parameter in the destination. Use the normal disk-buffer option if you want a solution that is faster than the reliable disk-buffer option. In this case, the process will be less reliable and it is possible to lose logs in case of AxoSyslog crash. The filename of the normal disk-buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.qf.
Example: Example for using normal disk-based buffering
For details on the differences between normal and reliable disk-based buffering, see also About disk queue files.
9.3.3 - How to get information about disk-buffer files
Purpose
This section describes how to get information about disk-buffer files used in AxoSyslog.
Note
While reading this section, consider that the default installation path used in the commands and AxoSyslog files is /opt/syslog-ng.
9.3.3.1 - Information about disk-buffer files
This section describes information about disk-buffer files used in AxoSyslog.
The following list contains information about how disk-buffer files are used in AxoSyslog:
You can configure disk-buffer() for a remote destination in the destination() statement.
For more information about an example of configuring disk-buffer() for a remote destination in the destination() statement, see disk-buffer().
By default, AxoSyslog creates disk-buffer files under /opt/syslog-ng/var directory, unless dir() option is set in disk-buffer().
The filenames are generated automatically by AxoSyslog with the extensions .qf for a normal disk-buffer and .rqf for a reliable disk-buffer.
The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination, but doesn’t store information about the destination.
9.3.3.2 - Getting the status information of disk-buffer files
Purpose
This section describes getting the status information of the disk-buffer files used in AxoSyslog.
Command syntax
The basic command syntax for getting the status information of the disk-buffer files used in AxoSyslog looks like the following:
/opt/syslog-ng/bin/dqtool info DISK-BUFFER_FILE
Example commands
The following example commands describe how you can get the status information of two different types of disk-buffer files (namely, empty normal disk-buffer files, and non-empty reliable disk-buffer queue files).
Example commands for empty, normal disk-buffer files, and non-empty, reliable disk-buffer queue files
Empty, normal disk-buffer file
/opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qfDisk-buffer state loaded;filename='/opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qf', number_of_messages='0'
Non-empty, reliable disk-buffer queue file
/opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.rqfReliable disk-buffer state loaded;filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', number_of_messages='10'
One-liner command to get the state of disk-buffer files in the default directory
You can use the following one-liner command to get the state of disk-buffer files in the default directory:
for qfile in /opt/syslog-ng/var/*.?(r)qf ;do /opt/syslog-ng/bin/dqtool info $qfile 2>&1;done
9.3.3.3 - Getting the list of disk-buffer files
Purpose
This section describes getting the list of disk-buffer files used in AxoSyslog.
The AxoSyslog application stores information (namely, the IP:PORT or DNS:PORT of the destinations, and the name of the disk-buffer file) about disk-buffer files in its persist file.
Example: command for listing the disk-buffer files in use
The following command will list the disk-buffer files in use:
This section describes orphan disk-buffer files used in AxoSyslog.
Orphan disk-buffer files
In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), AxoSyslog creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.
Note
The AxoSyslog application does not store messages in orphan disk-buffer files or forward the messages stored in the disk-buffer file.
Discovering the new disk-buffer files (orphan disk-buffer files)
To discover orphan disk-buffer files, get the list of disk-buffer files from the persist file, then compare the list with the contents of the disk-buffer files’ saving directory.
Example: difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files’ saving directory
The following examples show the difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files’ saving directory.
9.3.3.6 - How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance
Purpose
This section describes how to read messages from an orphan disk-buffer file by using a separate AxoSyslog process running parallel to the already running AxoSyslog instance.
Orphan disk-buffer files
In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), AxoSyslog creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.
Note
The AxoSyslog application does not store messages in orphan disk-buffer files or forward the messages stored in the disk-buffer file.
Processing the messages from an orphan disk-buffer file by using a separate AxoSyslog instance
When AxoSyslog creates orphan disk-buffer files, you can start a separate AxoSyslog instance parallel to the AxoSyslog instance already running, and use the following resolution process to process the messages in the orphan disk-buffer file.
Warning
Before starting a separate AxoSyslog instance to process the messages from the orphan disk-buffer file, consider the following:
During the resolution process, a separate AxoSyslog instance will be started with its temporary files beside the AxoSyslog instance already running.
An incorrect startup command and incorrect configurations may cause issues for the AxoSyslog instance already running.
The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination.
The disk-buffer file doesn’t store information about the destination.
To process the messages from an orphan disk-buffer file using a separate AxoSyslog instance,
It is important to know the type of the disk-buffer file. Disk-buffer file types can be normal (.qf) or reliable (.rqf).
In the examples during this process, the /opt/syslog-ng/var/syslog-ng-00005.rqf orphan reliable disk-buffer file is used.
Determine the destination of the logs. The content of the disk-buffer may help you determine the logs’ destination. For more information, see How to get information about disk-buffer files.
In the examples during this process, the destination 10.21.10.20 is used with the standard network() port 514.
Create a directory for the temporary instance. In the examples during this process, the /tmp/qdisk directory is used.
mkdir /tmp/qdisk
Warning
Make sure that there is sufficient disk space in the directory. The minimum recommended disk space in the directory is equal to the size of the orphan disk-buffer file.
Create the configuration file /tmp/qdisk/qdisk.conf for the temporary instance with the following content.
Example: creating the /tmp/qdisk/qdisk.conf configuration file for the temporary instance
Add your destination statement with disk-buffer() to the configuration file. You can copy the destination statement from your running AxoSyslog configuration.
Warning
Add the dir() option and set the disk-buffer file’s destination directory to the temporary directory (that is, /tmp/qdisk) in your destination statement.
Start the temporary AxoSyslog instance in the foreground.
The AxoSyslog application will log to the console, so you will see any potential error that may occur during startup.
The following example output displays that an empty disk-buffer file has been created and the connection to the remote destination has been established.
Example: output displaying newly created empty disk-buffer file and connection established to remote destination
Follow-mode file source not found, deferring open;filename='/no_such_file_or.dir' Reliable disk-buffer state saved;filename='/tmp/qdisk/syslog-ng-00000.rqf', qdisk_length='0' No server license found, running in client mode; syslog-ng starting up;version='7.0.20', cfg-fingerprint='eaa03b9efb88b87d7c1b0ce7efd042ed8ac0c013', cfg-nonce-ndx='0', cfg-signature='c0327a7f7e6418ce0399a75089377dfb662bb072' FIPS information; FIPS-mode='disabled' Syslog connection established;fd='7', server='AF_INET(10.21.10.20:514)', local='AF_INET(0.0.0.0:0)'
To stop AxoSyslog, press CTRL+C.
Overwrite the empty disk-buffer file with the orphan disk-buffer file.
If you have more than one orphan disk-buffer file, repeat the steps following the AxoSyslog stop (that is, the steps beginning from overwriting the empty disk-buffer file with the orphan disk-buffer file) for each orphan disk-buffer file.
Remove the temporary directory.
Example: command for removing the temporary directory
The following command removes the /mp/qdisk temporary directory:
rm -rf /tmp/qdisk
9.3.4 - Enabling memory buffering
To enable memory buffering, use the log-fifo-size() parameter in the destination. All destination drivers can use memory buffering. Use memory buffering if you want to send logs to destinations where disk-based buffering is not available. Or if you want the fastest solution, and if AxoSyslog crash or network downtime is never expected. In these cases, losing logs is possible. This solution does not use disk-based buffering, logs are stored only in the memory.
The key difference between disk queue files that employ the reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to AxoSyslog crashes. Of course, using the reliable(yes) option introduces a significant performance penalty as well.
Both reliable and normal disk-buffers employ an in-memory output queue (set in quot-size()) and an in-memory overflow queue (set in flow-control-window-bytes() for reliable disk-buffers, or flow-control-window-size() for normal disk-buffers). The difference between reliable and normal disk-buffers is that when the reliable disk-buffer uses one of its in-memory queues, it also stores the message on the disk, whereas the normal disk-buffer stores the message only in memory. The normal disk-buffer only uses the disk if the in-memory output buffer is filled up completely. This approach has better performance (due to fewer disk I/O operations), but also carries the risk of losing a maximum of quot-size() plus flow-control-window-size() number of messages in case of an unexpected power failure or application crash.
Size of the queue files
Disk queue files grow. Each may take up to capacity-bytes() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file.
The disk-buffer file’s size should be considered as the configured capacity-bytes() at any point of time, even if it does not have messages in it. Truncating the disk-buffer file can slow down disk I/O operations, so AxoSyslog does not always truncate the file when it would be possible (see the truncate-size-ratio() option). If a large disk-buffer file is not desirable, you should set the capacity-bytes() option to a smaller value. Note that AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default.
The disk-buffer file’s size does not strictly correlate to the number of stored messages. If you want to get information about the disk-buffer, use dqtool (for more information, see Getting the status information of disk-buffer files).
Note
If a queue file becomes corrupt, AxoSyslog starts a new one. This might lead to the queue files consuming more space in total than their maximal configured size and the number of configured queue files multiplied together.
Preallocating disk-buffer files
By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).
In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.
If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).
9.4 - Filters
The following sections describe how to select and filter log messages.
Using filters describes how to configure and use filters.
Tagging messages explains how to tag messages and how to filter on the tags.
Filter functions is a detailed description of the filter functions available in AxoSyslog.
9.4.1 - Using filters
Filters perform log routing: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.
To define a filter, add a filter statement to the syslog-ng.conf configuration file using the following syntax:
The following example does the same, but defines the filter inline.
log { source(s1); filter { host("example") and match("deny" value("MESSAGE"))}; destination(d1);};
9.4.2 - Combining filters with boolean operators
When a log statement includes multiple filter statements, AxoSyslog sends a message to the destination only if all filters are true for the message. In other words, the filters are connected with the logical AND operator. In the following example, no message arrives to the destination, because the filters are exclusive (the hostname of a client cannot be example1 and example2 at the same time):
Use the not operator to invert filters, for example, to select the messages that were not sent by host example1:
filter demo_filter { not host("example1");};
However, to select the messages that were not sent by host example1 or example2, you have to use the and operator (that’s how boolean logic works):
filter demo_filter { not host("example1") and not host("example2");};
Alternatively, you can use parentheses to avoid this confusion:
filter demo_filter { not (host("example1") or host("example2"));};
For a complete description on filter functions, see Filter functions.
The following filter statement selects the messages that contain the word deny and come from the host example.
filter demo_filter { host("example") and match("deny" value("MESSAGE"));};
The value() parameter of the match function limits the scope of the function to the text part of the message (that is, the part returned by the ${MESSAGE} macro), or optionally to the content of any other macro. The template() parameter of the match function can be used to run a filter against a combination of macros. For details on using the match() filter function, see match().
Note
Filters are often used together with log path flags. For details, see Log path flags.
9.4.3 - Comparing macro values in filters
In AxoSyslog you can compare macro values and templates as numerical and string values. String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string. For details on macros and templates, see Customize message format using macros and templates.
Use the following syntax to compare macro values or templates.
AxoSyslog versions earlier than 4.0 used separate operators for string comparisons (for example, eq). In version 4.0 and later, you can simply use the mathematical symbols as operators (like ==, !=, >=), and AxoSyslog automatically determines how to compare the arguments from their type. The logic behind that is similar to JavaScript:
If both sides of the comparisons are strings, then the comparison is string.
If one of the arguments is numeric, then the comparison is numeric.
Literal numbers (numbers not enclosed in quotes) are numeric.
You can explicitly type-cast an argument into a number.
For example:
if ("${.apache.httpversion}" == 1.0)
The right side of the == operator is 1.0, which is a floating point literal, so the comparison is numeric.
if (double("${.apache.httpversion}") == "1.0")
The left side is explicitly type cast into double, the right side is string (because of the quotes), so the comparison is numeric.
if ("${.apache.request}" == "/wp-admin/login.php")
The left side is not type-cast, so it’s a string, the right side is a string, so the comparison is string.
Note: You can still use the old string operators if you want to, they are available for backwards compatibility.
Example: Comparing macro values in filters
The following expression selects log messages containing a PID (that is, ${PID} macro is not empty):
filter f_pid {"${PID}" !=""};
The following expression selects log messages that do not contain a PID. Also, it uses a template as the left argument of the operator and compares the values as strings:
filter f_pid {"${HOST}${PID}"=="${HOST}"};
The following example selects messages with priority level higher than 5.
filter f_level {"${LEVEL_NUM}" > 5};
Make sure to:
Enclose macros and templates in double-quotes.
Use the $ character before macros.
Note that:
You can use type casting anywhere where you can use templates to apply a type to the result of the template expansion.
Using comparator operators can be equivalent to using filter functions, but is somewhat slower. For example, using "${HOST}" == "myhost" is equivalent to using host("myhost" type(string)).
You can use any macro in the expression, including user-defined macros from parsers and results of pattern database classifications.
The results of filter functions are boolean values, so they cannot be compared to other values.
You can use boolean operators to combine comparison expressions.
Comparison operators
The following numerical and string comparison operators are available.
Numerical or string operator
String operator
Meaning
==
eq
Equals
!=
ne
Not equal to
>
gt
Greater than
<
lt
Less than
>=
ge
Greater than or equal
=<
le
Less than or equal
9.4.4 - Using wildcards, special characters, and regular expressions in filters
The host(), match(), and program() filter functions accept regular expressions as parameters. The exact type of the regular expression to use can be specified with the type() option. By default, AxoSyslog uses PCRE regular expressions.
In regular expressions, the asterisk (\) character means 0, 1, or any number of the previous expression. For example, in the f\*ilter expression the asterisk means 0 or more f letters. This expression matches for the following strings: ilter, filter, ffilter, and so on. To achieve the wildcard functionality commonly represented by the asterisk character in other applications, use .\ in your expressions, for example, f.\*ilter.
Alternatively, if you do not need regular expressions, only wildcards, use type(glob) in your filter:
Example: Filtering with widcards
The following filter matches on hostnames starting with the myhost string, for example, on myhost-1, myhost-2, and so on.
To filter for special control characters like the carriage return (CR), use the \\r escape prefix in AxoSyslog version 3.0 and 3.1. In AxoSyslog 3.2 and later, you can also use the \\x escape prefix and the ASCII code of the character. For example, to filter on carriage returns, use the following filter:
filter f_carriage_return {match("\x0d" value ("MESSAGE"));};
9.4.5 - Tagging messages
You can label the messages with custom tags. Tags are simple labels, identified by their names, which must be unique. Currently AxoSyslog can tag a message at the following places:
At the source when the message is received.
AxoSyslog automatically adds the .source.<id_of_the_source_statement> tag to every incoming message.
You can add custom tag using the tags() option of the source.
AxoSyslog 4.7 and newer automatically adds the following tags if it encounters errors when parsing syslog messages.
message.utf8_sanitized
message.parse_error
syslog.missing_pri
syslog.missing_timestamp
syslog.invalid_hostname
syslog.unexpected_framing
syslog.rfc3164_missing_header
syslog.rfc5424_unquoted_sdata_value
When the message matches a pattern in the pattern database. For details on using the pattern database, see Using pattern databases, for details on creating tags in the pattern database, see The pattern database format.
Tags can be also added and deleted using rewrite rules. For details, see Adding and deleting tags.
You can use the tags() filter to select only specific messages.
Tagging messages and also filtering on the tags is very fast, much faster than other types of filters.
Tags are available locally, that is, if you add tags to a message on the client, these tags will not be available on the server.
To include the tags in the message, use the ${TAGS} macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the ${TAGS} macro in the .SDATA.meta part of the message. Note that the ${TAGS} macro is available only in AxoSyslog 3.1.1 and later.
Description: Match messages having one of the listed facility codes.
The facility() filter accepts both the name and the numerical code of the facility or the importance level. Facility codes 0-23 are predefined and can be referenced by their usual name. Facility codes above 24 are not defined.
You can use the facility filter the following ways:
Use a single facility name, for example, facility(user)
Use a single facility code, for example, facility(1)
Use a facility range (works only with facility names), for example, facility(local0..local5)
The AxoSyslog application recognizes the following facilities: (Note that some of these facilities are available only on specific platforms.)
syslog Message Facilities recognized by the facility() filter
Numerical Code
Facility name
Facility
0
kern
kernel messages
1
user
user-level messages
2
mail
mail system
3
daemon
system daemons
4
auth
security/authorization messages
5
syslog
messages generated internally by syslogd
6
lpr
line printer subsystem
7
news
network news subsystem
8
uucp
UUCP subsystem
9
cron
clock daemon
10
authpriv
security/authorization messages
11
ftp
FTP daemon
12
ntp
NTP subsystem
13
security
log audit
14
console
log alert
15
solaris-cron
clock daemon
16-23
local0..local7
locally used facilities (local0-local7)
9.4.6.2 - filter()
Synopsis:
filter(filtername)
Description: Call another filter rule and evaluate its value. For example:
filter demo_filter { host("example") and match("deny" value("MESSAGE"))}; filter inverted_demo_filter { not filter(demo_filter)}
9.4.6.3 - host()
Synopsis:
host(regexp)
Description: Match messages by using a regular expression against the hostname field of log messages. Note that you can filter only on the actual content of the HOST field of the message (or what it was rewritten to). That is, AxoSyslog will compare the filter expression to the content of the ${HOST} macro. This means that for the IP address of a host will not match, even if the IP address and the hostname field refers to the same host. To filter on IP addresses, use the netmask() filter.
filter demo_filter { host("example")};
9.4.6.4 - in-list()
Synopsis:
in-list("</path/to/file.list>", value(""))
Description: Matches the value of the specified field to a list stored in a file, allowing you to do simple, file-based black- and whitelisting. The file must be a plain-text file, containing one entry per line.
The AxoSyslog application loads the entire file, and compares the value of the specified field (for example, ${PROGRAM}) to entries in the file. When you use the in-list() filter, note the following points:
Comparing the values is case-sensitive.
Only exact matches are supported, partial and substring matches are not.
If you modify the list file, reload the configuration of AxoSyslog for the changes to take effect.
Make sure that there is an empty line in the file after the last entry. If the last entry is the last line, it will be excluded from the list.
Available in AxoSyslog 3.5 and later.
Example: Selecting messages using the in-list() filter
Create a text file that contains the programs (as in the ${PROGRAM} field of their log messages) you want to select. For example, you want to forward only the logs of a few applications from a host: kernel, sshd, and sudo. Create the /etc/syslog-ng/programlist.list file with the following contents. Note the empty line after the last entry.
kernel
sshd
sudo
The following filter selects only the messages of the listed applications:
Create the appropriate sources and destinations for your environment, then create a log path that uses the previous filter to select only the log messages of the applications you need:
To create a blacklist filter, simply negate the in-list() filter:
filter f_blacklist { not in-list("/etc/syslog-ng/programlist.list", value("PROGRAM"));};
9.4.6.5 - level() or priority()
Synopsis:
level() or level(..)
Description: The level() filter selects messages corresponding to a single importance level, or a level-range. To select messages of a specific level, use the name of the level as a filter parameter, for example, use the following to select warning messages:
level(warning)
To select a range of levels, include the beginning and the ending level in the filter, separated with two dots (..). For example, to select every message of error or higher level, use the following filter:
level(err..emerg)
The level() filter accepts the following levels: emerg, alert, crit, err, warning, notice, info, debug.
9.4.6.6 - match()
Synopsis:
match(regexp)
Description: Match a regular expression to the headers and the message itself (that is, the values returned by the MSGHDR and MSG macros). Note that in AxoSyslog version 2.1 and earlier, the match() filter was applied only to the text of the message, excluding the headers. This functionality has been moved to the message() filter.
To limit the scope of the match to a specific part of the message (identified with a macro), use the match(regexp value("MACRO")) syntax. Do not include the $ sign in the parameter of the value() option.
Starting with version 3.22, the match() filter can work on templates as well. This means that you can a match against an expression combined of macros, instead of a single macro. Note that when using a template, you must reference macros with the $ sign (unlike when using the value() parameter). For example:
Description: Match a regular expression to the text of the log message, excluding the headers (that is, the value returned by the MSG macros). Note that in syslog-ng version 2.1 and earlier, this functionality was performed by the match() filter.
9.4.6.8 - netmask()
Synopsis:
netmask(ipv4/mask)
Description: Select only messages sent by a host whose IP address belongs to the specified IPv4 subnet. Note that this filter checks the IP address of the last-hop relay (the host that actually sent the message to AxoSyslog), not the contents of the HOST field of the message. You can use both the dot-decimal and the CIDR notation to specify the netmask. For example, 192.168.5.0/255.255.255.0 or 192.168.5.0/24. To filter IPv6 addresses, see netmask6().
9.4.6.9 - netmask6()
Synopsis:
netmask6(ipv6/mask)
Description: Select only messages sent by a host whose IP address belongs to the specified IPv6 subnet. Note that this filter checks the IP address of the last-hop relay (the host that actually sent the message to AxoSyslog), not the contents of the HOST field of the message. You can use both the regular and the compressed format to specify the IP address, for example, 1080:0:0:0:8:800:200C:417A or 1080::8:800:200C:417A. If you do not specify the address, localhost is used.
Use the netmask (also called prefix) to specify how many of the leftmost bits of the address comprise the netmask (values 1-128 are valid). For example, the following specify a 60-bit prefix: 12AB:0000:0000:CD30:0000:0000:0000:0000/60 or 12AB::CD30:0:0:0:0/60. Note that if you set an IP address and a prefix, AxoSyslog will ignore the bits of the address after the prefix. To filter IPv4 addresses, see netmask().
The netmask6() filter is available in AxoSyslog 3.7 and later.
Warning
If the IP address is not syntactically correct, the filter will never match. The AxoSyslog application currently does not send a warning for such configuration errors.
9.4.6.10 - program()
Synopsis:
program(regexp)
Description: Match messages by using a regular expression against the program name field of log messages.
9.4.6.11 - rate-limit()
Synopsis:
rate-limit(key($HOST) rate(5000))
Description: Limits messages rate based on arbitrary keys in each message. The key will be resolved using the key() option. Each resolution will be allowed to have the number of messages each second, set by the rate() option. For example if key($HOST) and rate(5000) are set, and there are 2 hosts sending messages to AxoSyslog, a total of 10000 messages will be allowed by the rate-limit() filter, 5000 from the first and 5000 from the second host. If key() was not set instead, then 5000 messages would be allowed each second, regardless of their content.
Note
In AxoSyslog version 3.35 the rate-limit() filter was called throttle(). In AxoSyslog version 3.36 it got renamed to rate-limit(), but throttle() is still available for backward compatibility.
Note
Like every other filter, messages unmatched (outside of the rate limit) by the rate-limit() filter are dropped by default. Also, as every filter can be used in channels or if conditions, the messages unmatched can be caught and handled, like sent to a different destination, and so on.
Example: Using the rate-limit() filter
The following example depicts the scenario described in the description part of this section.
The rate-limit() filter has the following options.
key()
Type:
template
Default:
empty string
Description: The resolved template, that will be used to create unique rate-limit token buckets. In AxoSyslog version 4.4 and earlier, the name of this option was template().
rate()
Type:
number
Default:
N/A
Mandatory:
yes
Description: The number of messages for each unique macro resolution, that will be let through (matched) by the filter each second.
9.4.6.12 - source()
Synopsis:
source id
Description: Select messages of a source statement. This filter can be used in embedded log statements if the parent statement contains multiple source groups — only messages originating from the selected source group are sent to the destination of the embedded log statement.
9.4.6.13 - tags()
Synopsis:
tag
Description: Select messages labeled with the specified tag. Every message automatically has the tag of its source in .source.<id_of_the_source_statement> format. This option is available only in AxoSyslog 3.1 and later.
Example: Adding tags and filtering messages with tags
The AxoSyslog application automatically adds the class of the message as a tag using the .classifier.<message-class> format. For example, messages classified as “system” receive the .classifier.system tag. Use the tags() filter function to select messages of a specific class.
To skip the processing of a message without sending it to a destination, create a log statement with the appropriate filters, but do not include any destination in the statement, and use the final flag.
Example: Skipping messages
The following log statement drops all debug level messages without any further processing.
The syslog-ng application has a number of global options governing DNS usage, the timestamp format used, and other general points. Each option may have parameters, similarly to driver specifications. To set global options, add an options statement to the syslog-ng.conf configuration file using the following syntax:
The following options can be specified in the options statement, as described in Configuring global options.
bad-hostname()
Accepted values:
regular expression
Default:
no
Description: A regexp containing hostnames which should not be handled as hostnames.
chain-hostnames()
Accepted values:
yes, no
Default:
no
Description: Enable or disable the chained hostname format. If a client sends the log message directly to the AxoSyslog server, the chain-hostnames() option is enabled on the server, and the client sends a hostname in the message that is different from its DNS hostname (as resolved from DNS by the AxoSyslog server), then the server can append the resolved hostname to the hostname in the message (separated with a / character) when the message is written to the destination.
For example, consider a client-server scenario with the following hostnames: client-hostname-from-the-message, client-hostname-resolved-on-the-server, server-hostname. The hostname of the log message written to the destination depends on the keep-hostname() and the chain-hostnames() options. How keep-hostname() and chain-hostnames() options are related is described in the following table.
If the log message is forwarded to the AxoSyslog server via a AxoSyslog relay, the hostname depends on the settings of the keep-hostname() and the chain-hostnames() options both on the AxoSyslog relay and the AxoSyslog server.
For example, consider a client-relay-server scenario with the following hostnames: client-hostname-from-the-message, client-hostname-resolved-on-the-relay, client-hostname-resolved-on-the-server, relay-hostname-resolved-on-the-server. How keep-hostname() and chain-hostnames() options are related is described in the following table.
The chain-hostnames() option can interfere with the way AxoSyslog counts the log source hosts. As a result, AxoSyslog falsely perceives several hosts logging to the central server, especially if the clients sends a hostname in the message that is different from its real hostname (as resolved from DNS). Disable the chain-hostnames() option on your log sources to avoid any problems related to license counting.
check-hostname()
Accepted values:
yes, no
Default:
no
Description: Enable or disable checking whether the hostname contains valid characters.
create-dirs()
Accepted values:
yes, no
Default:
no
Description: Enable or disable directory creation for destination files and sockets.
custom-domain()
Note
This global option works only if the use-fqdn() global option is set to yes.
Accepted values:
string
Default:
empty string
Description: Use this option to specify a custom domain name that is appended after the short hostname to receive the fully qualified domain name (FQDN). This option affects every outgoing message: eventlog sources, file sources, MARK messages and internal messages of AxoSyslog.
If the hostname is a short hostname, the custom domain name is appended after the hostname (for example, mypc becomes mypc.customcompany.local).
If the hostname is an FQDN, the domain name part is replaced with the custom domain name (for example, if the FQDN in the forwarded message is mypc.mycompany.local and the custom domain name is customcompany.local, the hostname in the outgoing message becomes mypc.customcompany.local).
dir-group()
Accepted values:
groupid
Default:
root
Description: The default group for newly created directories.
dir-owner()
Accepted values:
userid
Default:
root
Description: The default owner of newly created directories.
dir-perm()
Accepted values:
permission value
Default:
-1
Description: The permission mask of directories created by syslog-ng. Log directories are only created if a file after macro expansion refers to a non-existing directory, and directory creation is enabled (see also the create-dirs() option). For octal numbers prefix the number with 0, for example, use 0755 for rwxr-xr-x.
To preserve the original properties of an existing directory, use the option without specifying an attribute: dir-perm(). Note that when creating a new directory without specifying attributes for dir-perm(), the default permission of the directories is masked with the umask of the parent process (typically 0022).
Starting with version 3.16, the default value of this option is -1, so AxoSyslog does not change the ownership, unless explicitly configured to do so.
dns-cache()
Accepted values:
yes, no
Default:
yes
Description: Enable or disable DNS cache usage.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
dns-cache-expire()
Accepted values:
number
Default:
3600
Description: Number of seconds while a successful lookup is cached.
dns-cache-expire-failed()
Accepted values:
number
Default:
60
Description: Number of seconds while a failed lookup is cached.
dns-cache-hosts()
Accepted values:
filename
Default:
unset
Description: Name of a file in /etc/hosts format that contains static IP->hostname mappings. Use this option to resolve hostnames locally without using a DNS. Note that any change to this file triggers a reload in syslog-ng and is instantaneous.
dns-cache-size()
Accepted values:
number of hostnames
Default:
1007
Description: Number of hostnames in the DNS cache.
file-template()
Accepted values:
string
Default:
Description: Specifies a template that file-like destinations use by default. For example:
Description: Specifies how many lines are flushed to a destination at a time. The AxoSyslog application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.
The AxoSyslog application flushes the messages if it has sent flush-lines() number of messages, or the queue became empty. If you stop or reload AxoSyslog or in case of network sources, the connection with the client is closed, AxoSyslog automatically sends the unsent messages to the destination.
frac-digits()
Type:
number
Default:
0
Description: The AxoSyslog application can store fractions of a second in the timestamps according to the ISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received.
Note
The AxoSyslog application can add the fractions to non-ISO8601 timestamps as well.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
group()
Accepted values:
groupid
Default:
root
Description: The default group of output files. By default, syslog-ng changes the privileges of accessed files (for example, /dev/null) to root.root 0600. To disable modifying privileges, use this option with the -1 value.
jvm-options()
Type:
list
Default:
N/A
Description: Specify the Java Virtual Machine (JVM) settings of your Java destination from the AxoSyslog configuration file.
For example:
jvm-options("-Xss1M -XX:+TraceClassLoading")
keep-hostname()
Type:
yes or no
Default:
no
Description: Enable or disable hostname rewriting.
If enabled (keep-hostname(yes)), AxoSyslog assumes that the incoming log message was sent by the host specified in the HOST field of the message.
If disabled (keep-hostname(no)), AxoSyslog rewrites the HOST field of the message, either to the IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to AxoSyslog. For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.
Note
If the log message does not contain a hostname in its HOST field, AxoSyslog automatically adds a hostname to the message.
For messages received from the network, this hostname is the address of the host that sent the message (this means the address of the last hop if the message was transferred via a relay).
For messages received from the local host, AxoSyslog adds the name of the host.
This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
When relaying messages, enable this option on the AxoSyslog server and also on every relay, otherwise AxoSyslog will treat incoming messages as if they were sent by the last relay.
keep-timestamp()
Type:
yes or no
Default:
yes
Description: Specifies whether AxoSyslog should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
log-fifo-size()
Accepted values:
number (messages)
Default:
10000
Description: The number of messages that the output queue can store.
log-level()
Accepted values:
default, verbose, debug, trace
Default:
default
Description: Controls AxoSyslog’s own internal log level. Corresponds to setting the internal log level using syslog-ng-ctl or the command line options of syslog-ng (the -d, -v, and -t ). Setting the log level in the configuration makes it easier to control logging in containerized environments where changing command line options is more problematic.
Available in AxoSyslog 4.0 and later.
Higher log-levels automatically include messages from lower log-levels:
default: Just normal log messages.
verbose: Normal and verbose log messages.
debug: Include debug messages of AxoSyslog.
trace: Include trace messages of how messages are processed.
options { log-level(debug);};
log-msg-size()
Accepted values:
number (bytes)
Default:
65536
Description: Maximum length of an incoming message in bytes. This length includes the entire message (the data structure and individual fields). The maximal value that can be set is 268435456 bytes (256 MiB).
For messages using the IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64 KiB.
Note
In most cases, <code>log-msg-size()</code> does not need to be set higher than 10 MiB.
Description: The mark-freq() option is an alias for the deprecated mark() option. This is retained for compatibility with AxoSyslog version 1.6.x.
mark-freq()
Accepted values:
number [seconds]
Default:
1200
Description: An alias for the obsolete mark() option, retained for compatibility with version 1.6.x.
The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() is periodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq() will be inherited from the global options. If the destination uses internal mark-mode(), then the global mark-freq() will be valid (does not matter what mark-freq() set in the destination side).
Description: The mark-mode() option can be set for the following destination drivers: file(), program(), unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.
internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yields the mark messages from internal source. This is the mode as AxoSyslog 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:
dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
host-idle: Sends MARK signal if there was NO local message on destination drivers. for example, MARK is generated even if messages were received from tcp. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signal from internal source will be dropped.
MARK signal can be sent by the following destination drivers: network(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().
none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.
global: Destination driver uses the global mark-mode() setting. Note that setting the global mark-mode() to global causes a syntax error in AxoSyslog.
Note
In case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is not open yet.
Available in AxoSyslog 3.4 and later.
normalize-hostnames()
Accepted values:
yes, no
Default:
no
Description: If enabled (normalize-hostnames(yes)), AxoSyslog converts the hostnames to lowercase.
Note
This setting applies only to hostnames resolved from DNS. It has no effect if the keep-hostname() option is enabled, and the message contains a hostname.
Description: Controls what happens when type-casting fails and AxoSyslog cannot convert some data to the specified type. By default, AxoSyslog drops the entire message and logs the error. Currently the value-pairs() option uses the settings of on-error().
drop-message: Drop the entire message and log an error message to the internal() source. This is the default behavior of AxoSyslog.
drop-property: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the internal() source.
fallback-to-string: Convert the property to string and log an error message to the internal() source.
silently-drop-message: Drop the entire message silently, without logging the error.
silently-drop-property: Omit the affected property (macro, template, or message-field) silently, without logging the error.
silently-fallback-to-string: Convert the property to string silently, without logging the error.
owner()
Accepted values:
userid
Default:
root
Description: The default owner of output files. If set, syslog-ng changes the owner of accessed files (for example, /dev/null) to this value, and the permissions to the value set in the perm() option.
Starting with version 3.16, the default value of this option is -1, so AxoSyslog does not change the ownership, unless explicitly configured to do so.
pass-unix-credentials()
Accepted values:
yes
Default:
yes
Description: Enable AxoSyslog to collect UNIX credential information (that is, the PID, user ID, and group of the sender process) for messages received using UNIX domain sockets. Available only in AxoSyslog 3.7 and later. Note that collecting UNIX credential information from sockets in high-traffic environments can be resource intensive, therefore pass-unix-credentials() can be disabled globally, or separately for each source.
perm()
Accepted values:
permission value
Default:
0600
Description: The default permission for output files. If set, syslog-ng changes the permissions of accessed files (for example, /dev/null) to this value, and the onwer to the value set in the owner() option.
Starting with version 3.16, the default value of this option is -1, so AxoSyslog does not change the permissions, unless explicitly configured to do so.
proto-template()
Accepted values:
name of a template
Default:
The default message format of the used protocol
Description: Specifies a template that protocol-like destinations (for example, network() and syslog()) use by default. For example:
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
send-time-zone()
Accepted values:
name of the timezone, or the timezone offset
Default:
local timezone
Description: Specifies the time zone associated with the messages sent by syslog-ng, if not specified otherwise in the message or in the destination driver. For details, see Timezones and daylight saving.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
stats()
Available in AxoSyslog 4.1 and later.
Description: The stats() option is a collection of statistics-related options.
Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Set to 0 to disable the STATS messages.
level()
Accepted values:
0
Default:
0
Description: Specifies the detail of statistics AxoSyslog collects about the processed messages.
Level 0 collects only statistics about the sources and destinations.
Level 1 contains details about the different connections and log files, but has a slight memory overhead.
Level 2 contains detailed statistics based on the hostname.
Level 3 contains detailed statistics based on various message parameters like facility, severity, or tags.
Note that level 2 and 3 increase the memory requirements and CPU load. For details on message statistics, see Statistics of AxoSyslog.
max-dynamics()
Accepted values:
number
Default:
N/A
Available in AxoSyslog 4.1 and later.
Description: To avoid performance issues or even overloading AxoSyslog (for example, if a script starts to send logs from different IP addresses to AxoSyslog), you might want to limit the number of registered dynamic counters in the message statistics. For details on message statistics, see Statistics of AxoSyslog.
Unlimited dynamic counters:
If you do not use this option, dynamic counters will not be limited. This can be useful in cases where you are extremely interested in dynamic counters, and use these statistics extensively.
Warning
In some cases, there might be even millions of dynamic counters
Limited dynamic counter clusters:
To limit dynamic counters, enter a number, and only a maximum of <number> counters will be registered in the statistics.
In practice, this means dynamic counter clusters. A program name produces one dynamic counter cluster, that can include several counters, such as processed, stamp, and so on.
Example: Limiting dynamic counter clusters 1:
If you set stats-max-dynamics() to 1, and 2 programs send messages, only one of these programs will be tracked in the dynamic counters, but it will have more than one counters.
Example: Limiting dynamic counter clusters 2:
If you have 500 clients, and set stats-max-dynamics() to 1000, you will have enough number of counters reserved for these clients, but at the same time, you limit the use of your resources and therefore protect your system from being overloaded.
No dynamic counters:
To disable dynamic counters completely, set the value of this option to 0. This is the recommended value if you do not use statistics, or if you are not interested in dynamic counters in particular (for example, the number of logs arriving from programs).
Note
If you set a lower value to stats-max-dynamics() (or, any limiting value, if this option has not been configured before) and restart AxoSyslog, the changes will only be applied after stats-freq() time has passed. That is, the previously allocated dynamic clusters will only be removed after this time.
syslog-stats()
Accepted values:
yes, no, auto
Default:
auto
Available in AxoSyslog 4.1 and later.
Description: Changes the behavior of counting messages based on different syslog fields, like SEVERITY, FACILITY, HOST.
Description: Enable AxoSyslog to run in multithreaded mode and use multiple CPUs. Available only in AxoSyslog 3.3 and later. Note that setting threaded(no) does not mean that AxoSyslog will use only a single thread. For details, see Multithreading and scaling.
time-reap()
Accepted values:
number (seconds)
Default:
60 or 0, see description for details
Description: The time to wait in seconds before an idle destination file or pipe is closed. Note that only destination files having macros in their filenames are closed automatically.
Starting with version 3.23, the way how time-reap() works is the following.
If the time-reap() option of the destination is set, that value is used, for example:
destination d_fifo { pipe("/tmp/test.fifo",
time-reap(30)# sets time-reap() for this destination only);};
If the time-reap() option of the destination is not set, and the destination does not use a template or macro in its filename or path, time-reap() is automatically set to 0. For example:
destination d_fifo { pipe("/tmp/test.fifo",
);};
Otherwise, the value of the global time-reap() option is used, which defaults to 60 seconds.
time-reopen()
Accepted values:
number [seconds]
Default:
60
Description: The time to wait in seconds before a dead connection is reestablished.
time-sleep() (DEPRECATED)
Accepted values:
number
Default:
0
Description: The time to wait in milliseconds between each invocation of the poll() iteration.
time-zone()
Type:
name of the timezone, or the timezone offset
Default:
unspecified
Description: Convert timestamps to the timezone specified by this option. If this option is not set, then the original timezone information in the message is used. Converting the timezone changes the values of all date-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, see Date-related macros.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
trim-large-messages()
Accepted values:
yes
Default:
no
Description: Determines what AxoSyslog does with incoming log messages that are received using the IETF-syslog protocol using the syslog() driver, and are longer than the value of log-msg-size(). Other drivers ignore the trim-large-messages() option.
If set to no, AxoSyslog drops the incoming log message.
If set to yes, AxoSyslog trims the incoming log message to the size set in log-msg-size(), and adds the trimmed tag to the message. The rest of the message is dropped. You can use the tag to filter on such messages.
filter f_trimmed { tags("trimmed");};
If AxoSyslog trims a log message, it sends a debug-level log message to its internal() source.
As a result of trimming, a parser could fail to parse the trimmed message. For example, a trimmed JSON or XML message will not be valid JSON or XML.
Available in AxoSyslog version 3.21 and later.
ts-format()
Accepted values:
rfc3164
Default:
rfc3164
Description: Specifies the timestamp format used when AxoSyslog itself formats a timestamp and nothing else specifies a format (for example: STAMP macros, internal messages, messages without original timestamps). For details, see also A note on timezones and timestamps.
By default, timestamps include only seconds. To include fractions of a second (for example, milliseconds) use the frac-digits() option.
Note
This option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(), or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use the proto-template option.
use-dns()
Type:
yes, no, persist_only
Default:
yes
Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example, from /etc/hosts). The AxoSyslog application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your AxoSyslog network endpoint with firewall rules, and make sure that all hosts which may get to AxoSyslog are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
use-fqdn()
Type:
yes or no
Default:
no
Description: Use this option to add a Fully Qualified Domain Name (FQDN) instead of a short hostname. You can specify this option either globally or per-source. The local setting of the source overrides the global option if available.
Note
Set use-fqdn() to yes if you want to use the custom-domain() global option.
Note
This option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
use-rcptid()
Accepted values:
`yes
Default:
no
Description: When the use-rcptid global option is set to yes, AxoSyslog automatically assigns a unique reception ID to every received message. You can access this ID and use it in templates via the ${RCPTID} macro. The reception ID is a monotonously increasing 48-bit integer number, that can never be zero (if the counter overflows, it restarts with 1).
This option is deprecated, use the use-uniqid() option instead.
use-uniqid()
Accepted values:
yes, no
Default:
no
Description: This option enables generating a globally unique ID. It is generated from the HOSTID and the RCPTID in the format of HOSTID@RCPTID. It has a fixed length: 16+@+8 characters. You can include the unique ID in the message by using the macro. For details, see UNIQID.
Enabling this option automatically generates the HOSTID. The HOSTID is a persistent, 32-bits-long cryptographically secure pseudo random number, that belongs to the host that the AxoSyslog is running on. If the persist file is damaged, the HOSTID might change.
Enabling this option automatically enables the RCPTID functionality. For details, see RCPTID
11 - TLS-encrypted message transfer
11.1 - Secure logging using TLS
The AxoSyslog application can send and receive log messages securely over the network using the Transport Layer Security (TLS) protocol using the network() and syslog() drivers.
Note
This chapter describes how to use TLS encryption when using the standard syslog protocols, that is, the network() and syslog() drivers, for example, to forward log messages between two AxoSyslog nodes, or to send log data to a log server. Other destinations that support TLS-encryption are not discussed in this chapter (for example, http()).
TLS uses certificates to authenticate and encrypt the communication, as illustrated on the following figure:
The client authenticates the server by requesting its certificate and public key. Optionally, the server can also request a certificate from the client, thus mutual authentication is also possible.
In order to use TLS encryption in syslog-ng, the following elements are required:
A certificate on the AxoSyslog server that identifies the server.
The certificate of the Certificate Authority that issued the certificate of the AxoSyslog server (or the self-signed certificate of the AxoSyslog server) must be available on the AxoSyslog client.
When using mutual authentication to verify the identity of the clients, the following elements are required:
A certificate must be available on the AxoSyslog client. This certificate identifies the AxoSyslog client.
The certificate of the Certificate Authority that issued the certificate of the AxoSyslog client must be available on the AxoSyslog server.
Mutual authentication ensures that the AxoSyslog server accepts log messages only from authorized clients.
For more information about TLS-related error messages, see Error messages.
11.2 - Encrypting log messages with TLS
This section describes how to configure TLS encryption in syslog-ng. For the concepts of using TLS in syslog-ng, see Secure logging using TLS.
11.2.1 - Configuring TLS on the AxoSyslog clients
Purpose:
Complete the following steps on every AxoSyslog client host. Examples are provided using both the legacy BSD-syslog protocol (using the network() driver) and the new IETF-syslog protocol standard (using the syslog() driver):
Steps:
Copy the CA certificate (for example, cacert.pem) of the Certificate Authority that issued the certificate of the AxoSyslog server (or the self-signed certificate of the AxoSyslog server) to the AxoSyslog client hosts, for example, into the /opt/syslog-ng/etc/syslog-ng/ca.d directory.
Issue the following command on the certificate: openssl x509 -noout -hash -in cacert.pem The result is a hash (for example, 6d2962a8), a series of alphanumeric characters based on the Distinguished Name of the certificate.
Issue the following command to create a symbolic link to the certificate that uses the hash returned by the previous command and the .0 suffix.
ln -s cacert.pem 6d2962a8.0
Add a destination statement to the syslog-ng.conf configuration file that uses the tls( ca-dir(path_to_ca_directory) ) option and specify the directory using the CA certificate. The destination must use the network() or the syslog() destination driver, and the IP address and port parameters of the driver must point to the AxoSyslog server.
Example: A destination statement using TLS
The following destination encrypts the log messages using TLS and sends them to the 6514/TCP port of the AxoSyslog server having the 10.1.2.3 IP address.
Include the destination created in Step 2 in a log statement.
Warning
The encrypted connection between the server and the client fails if the Common Name or the subject_alt_name parameter of the server certificate does not contain the hostname or the IP address (as resolved from the AxoSyslog clients and relays) of the server.
Do not forget to update the certificate files when they expire.
11.2.2 - Configuring TLS on the AxoSyslog server
Purpose:
Complete the following steps on the AxoSyslog server:
Steps:
Create an X.509 certificate for the AxoSyslog server.
Note
The subject_alt_name parameter (or the Common Name parameter if the subject_alt_name parameter is empty) of the server’s certificate must contain the hostname or the IP address (as resolved from the syslog-ng clients and relays) of the server (for example, syslog-ng.example.com).
Alternatively, the Common Name or the subject_alt_name parameter can contain a generic hostname, for example, \*.example.com.
Note that if the Common Name of the certificate contains a generic hostname, do not specify a specific hostname or an IP address in the subject_alt_name parameter.
Copy the certificate (for example, syslog-ng.cert) of the AxoSyslog server to the AxoSyslog server host, for example, into the /opt/syslog-ng/etc/syslog-ng/cert.d directory. The certificate must be a valid X.509 certificate in PEM format.
Copy the private key (for example, syslog-ng.key) matching the certificate of the AxoSyslog server to the AxoSyslog server host, for example, into the /opt/syslog-ng/etc/syslog-ng/key.d directory. The key must be in PEM format. If you want to use a password-protected key, see Password-protected keys.
Add a source statement to the syslog-ng.conf configuration file that uses the tls( key-file(key_file_fullpathname) cert-file(cert_file_fullpathname) ) option and specify the key and certificate files. The source must use the source driver (network() or syslog()) matching the destination driver used by the AxoSyslog client.
Example: A source statement using TLS
The following source receives log messages encrypted using TLS, arriving to the 1999/TCP port of any interface of the AxoSyslog server.
Disable mutual authentication for the source by setting the following TLS option in the source statement: tls( peer-verify(optional-untrusted);
If you want to authenticate the clients, you have to configure mutual authentication. For details, see Mutual authentication using TLS.
For the details of the available tls() options, see TLS options.
Example: Disabling mutual authentication
The following source receives log messages encrypted using TLS, arriving to the 1999/TCP port of any interface of the AxoSyslog server. The identity of the AxoSyslog client is not verified.
Do not forget to update the certificate and key files when they expire.
11.3 - Mutual authentication using TLS
This section describes how to configure mutual authentication between the AxoSyslog server and the client. Configuring mutual authentication is similar to configuring TLS (for details, see Encrypting log messages with TLS), but the server verifies the identity of the client as well. Therefore, each client must have a certificate, and the server must have the certificate of the CA that issued the certificate of the clients. For the concepts of using TLS in syslog-ng, see Secure logging using TLS.
11.3.1 - Configuring TLS on the AxoSyslog clients
Purpose:
Complete the following steps on every AxoSyslog client host. Examples are provided using both the legacy BSD-syslog protocol (using the network() driver) and the new IETF-syslog protocol standard (using the syslog() driver):
Steps:
Create an X.509 certificate for the AxoSyslog client.
Copy the certificate (for example, client_cert.pem) and the matching private key (for example, client.key) to the AxoSyslog client host, for example, into the /opt/syslog-ng/etc/syslog-ng/cert.d directory. The certificate must be a valid X.509 certificate in PEM format. If you want to use a password-protected key, see Password-protected keys.
Copy the CA certificate of the Certificate Authority (for example, cacert.pem) that issued the certificate of the AxoSyslog server (or the self-signed certificate of the syslog-ng server) to the AxoSyslog client hosts, for example, into the /opt/syslog-ng/etc/syslog-ng/ca.d directory.
Issue the following command on the certificate: openssl x509 -noout -hash -in cacert.pem The result is a hash (for example, 6d2962a8), a series of alphanumeric characters based on the Distinguished Name of the certificate.
Issue the following command to create a symbolic link to the certificate that uses the hash returned by the previous command and the .0 suffix.
ln -s cacert.pem 6d2962a8.0
Add a destination statement to the syslog-ng.conf configuration file that uses the tls( ca-dir(path_to_ca_directory) ) option and specify the directory using the CA certificate. The destination must use the network() or the syslog() destination driver, and the IP address and port parameters of the driver must point to the AxoSyslog server. Include the client’s certificate and private key in the tls() options.
Example: A destination statement using mutual authentication
The following destination encrypts the log messages using TLS and sends them to the 1999/TCP port of the AxoSyslog server having the 10.1.2.3 IP address. The private key and the certificate file authenticating the client is also specified.
Include the destination created in Step 2 in a log statement.
Warning
The encrypted connection between the server and the client fails if the Common Name or the subject_alt_name parameter of the server certificate does not match the hostname or the IP address (as resolved from the AxoSyslog clients and relays) of the server.
Do not forget to update the certificate files when they expire.
11.3.2 - Configuring TLS on the AxoSyslog server
Purpose:
Complete the following steps on the AxoSyslog server:
Steps:
Copy the certificate (for example, syslog-ng.cert) of the AxoSyslog server to the AxoSyslog server host, for example, into the /opt/syslog-ng/etc/syslog-ng/cert.d directory. The certificate must be a valid X.509 certificate in PEM format.
Copy the CA certificate (for example, cacert.pem) of the Certificate Authority that issued the certificate of the AxoSyslog clients to the AxoSyslog server, for example, into the /opt/syslog-ng/etc/syslog-ng/ca.d directory.
Issue the following command on the certificate: openssl x509 -noout -hash -in cacert.pem The result is a hash (for example, 6d2962a8), a series of alphanumeric characters based on the Distinguished Name of the certificate.
Issue the following command to create a symbolic link to the certificate that uses the hash returned by the previous command and the .0 suffix.
ln -s cacert.pem 6d2962a8.0
Copy the private key (for example, syslog-ng.key) matching the certificate of the AxoSyslog server to the AxoSyslog server host, for example, into the /opt/syslog-ng/etc/syslog-ng/key.d directory. The key must be in PEM format. If you want to use a password-protected key, see Password-protected keys.
Add a source statement to the syslog-ng.conf configuration file that uses the tls( key-file(key_file_fullpathname) cert-file(cert_file_fullpathname) ) option and specify the key and certificate files. The source must use the source driver (network() or syslog()) matching the destination driver used by the syslog-ng client. Also specify the directory storing the certificate of the CA that issued the client’s certificate.
For the details of the available tls() options, see TLS options.
Example: A source statement using TLS
The following source receives log messages encrypted using TLS, arriving to the 1999/TCP port of any interface of the AxoSyslog server.
Do not forget to update the certificate and key files when they expire.
11.4 - Password-protected keys
Starting with AxoSyslog version 3.14, you can use password-protected private keys in the network() and syslog() source and destination drivers.
Restrictions and limitations
Note
Hazard of data loss! If you use password-protected keys, you must provide the passphrase of the password-protected keys every time AxoSyslog is restarted (AxoSyslog keeps the passphrases over reloads). The sources and destinations that use these keys will not work until you provide the passwords. Other parts of the AxoSyslog configuration will be unaffected.
This means that if you use a password-protected key in a destination, and you use this destination in a log path that has multiple destinations, neither destinations will receive log messages until you provide the password. In this cases, always use disk-based buffering to avoid data loss.
The path and the filename of the private key cannot contain whitespaces.
Depending on your platform, the number of passwords AxoSyslog can use at the same time might be limited (for example, on Ubuntu 16.04 you can store 16 passwords if you are running AxoSyslog as a non-root user). If you use lots of password-protected private keys in your AxoSyslog configuration, increase this limit using the following command: sudo ulimit -l unlimited
Providing the passwords
The syslog-ng-ctl credentials status command allows you to query the status of the private keys that AxoSyslog uses in the network() and syslog() drivers. The command returns the list of private keys used, and their status. For example:
syslog-ng-ctl credentials status
Secret store status:
/home/user/ssl_test/client-1/client-encrypted.key SUCCESS
If the status of a key is PENDING, you must provide the passphrase for the key, otherwise AxoSyslog cannot use it. The sources and destinations that use these keys will not work until you provide the passwords. Other parts of the AxoSyslog configuration will be unaffected. You must provide the passphrase of the password-protected keys every time AxoSyslog is restarted.
The following log message also notifies you of PENDING passphrases:
Waiting for password;keyfile='private.key'
You can add the passphrase to a password-protected private key file using the following command. AxoSyslog will display a prompt for you to enter the passphrase. We recommend that you use this method.
This page describes the TLS-related options of the network() and syslog() drivers. Where applicable, other drivers also support encrypted transport, see the documentation of the other drivers for details.
Note
The format of the TLS connections used by AxoSyslog is similar to using AxoSyslog and stunnel, but the source IP information is not lost.
To encrypt connections, use the transport("tls") and tls() options in the source and destination statements.
The tls() option can include the following settings:
allow-compress()
Accepted values:
`yes
Default:
no
Description: Enable on-the-wire compression in TLS communication. Note that this option must be enabled both on the server and the client to have any effect. Enabling compression can significantly reduce the bandwidth required to transport the messages, but can slightly decrease the performance of AxoSyslog, reducing the number of transferred messages during a given period.
Available in version 3.19 and later.
ca-dir()
Accepted values:
Directory name
Default:
none
Description: The name of a directory that contains a set of trusted CA certificates in PEM format. The CA certificate files have to be named after the 32-bit hash of the subject’s name. This naming can be created using the c_rehash utility in openssl. For an example, see Configuring TLS on the AxoSyslog clients. The AxoSyslog application uses the CA certificates in this directory to validate the certificate of the peer.
This option can be used together with the optional ca-file() option.
ca-file()
Accepted values:
File name
Default:
empty
Description: Optional. The name of a file that contains a set of trusted CA certificates in PEM format. The AxoSyslog application uses the CA certificates in this file to validate the certificate of the peer.
Example format in configuration:
ca-file("/etc/pki/tls/certs/ca-bundle.crt")
Note
The ca-file() option can be used together with the ca-dir() option, and it is relevant when peer-verify() is set to other than no or optional-untrusted.
cert-file()
Accepted values:
Filename
Default:
none
Description: Name of a file, that contains an X.509 certificate (or a certificate chain) in PEM format, suitable as a TLS certificate, matching the private key set in the key-file() option. The AxoSyslog application uses this certificate to authenticate the AxoSyslog client on the destination server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
cipher-suite()
Accepted values:
Name of a cipher, or a colon-separated list
Default:
Depends on the OpenSSL version that AxoSyslog uses
Description: Specifies the cipher, hash, and key-exchange algorithms used for the encryption, for example, ECDHE-ECDSA-AES256-SHA384. The list of available algorithms depends on the version of OpenSSL used to compile AxoSyslog. To specify multiple ciphers, separate the cipher names with a colon, and enclose the list between double-quotes, for example:
For a list of available algorithms, execute the openssl ciphers -v command. The first column of the output contains the name of the algorithms to use in the cipher-suite() option, the second column specifies which encryption protocol uses the algorithm (for example, TLSv1.2). That way, the cipher-suite() also determines the encryption protocol used in the connection: to disable SSLv3, use an algorithm that is available only in TLSv1.2, and that both the client and the server supports. You can also specify the encryption protocols using ssl-options().
You can also use the following command to automatically list only ciphers permitted in a specific encryption protocol, for example, TLSv1.2:
echo"cipher-suite(\"$(openssl ciphers -v | grep TLSv1.2 | awk '{print $1}'| xargs echo -n | sed 's/ /:/g'| sed -e 's/:$//')\")"
Note that starting with version 3.10, when AxoSyslog receives TLS-encrypted connections, the order of ciphers set on the AxoSyslog server takes precedence over the client settings.
client-sigalgs()
Accepted values:
string [colon-separated list]
Default:
none
Description: A colon-separated list that specifies the supported signature algorithms associated with client authentication for TLSv1.2 and higher, for example, RSA-PSS+SHA256:ed25519.
For servers, the value is used in the signature_algorithms field of a CertificateRequest message.
For clients, it is used to determine which signature algorithm to use with the client certificate.
If client-sigalgs() is not set but sigalgs() is, then the values of sigalgs() are used.
crl-dir()
Accepted values:
Directory name
Default:
none
Description: Name of a directory that contains the Certificate Revocation Lists for trusted CAs. Similarly to ca-dir() files, use the 32-bit hash of the name of the issuing CAs as filenames. The extension of the files must be .r0.
dhparam-file()
Accepted values:
string (filename)
Default:
none
Description: Specifies a file containing Diffie-Hellman parameters, generated using the openssl dhparam utility. Note that AxoSyslog supports only DH parameter files in the PEM format. If you do not set this parameter, AxoSyslog uses the 2048-bit MODP Group, as described in RFC 3526.
ecdh-curve-list()
Accepted values:
string [colon-separated list]
Default:
none
Description: A colon-separated list that specifies the curves that are permitted in the connection when using Elliptic Curve Cryptography (ECC).
This option is only available when syslog-ng is compiled with OpenSSL version 1.0.2 or later. In the case of older versions, prime256v1 (NIST P-256) is used.
The following example curves work for all versions of OpenSSL that are equal to or later than version 1.0.2:
ecdh-curve-list("prime256v1:secp384r1")
key-file()
Accepted values:
Filename
Default:
none
Description: The name of a file that contains an unencrypted private key in PEM format, suitable as a TLS key. If properly configured, the AxoSyslog application uses this private key and the matching certificate (set in the cert-file() option) to authenticate the AxoSyslog client on the destination server.
keylog-file()
Accepted values:
Filename
Default:
N/A
Description: This option enables saving TLS secrets (decryption keys) for a given source or destination, which can be used to decrypt data with, for example, Wireshark. The given path and name of a file will be used to save these secrets.
This option is only available with the following drivers:
network
syslog
tcp
tcp6
Warning
Using keylog-file() makes TLS connections less secure by writing secret key materials into the given file. This option should only be enabled for debugging purposes and should be disabled after that. It is also recommended to delete the file after the debugging session is over.
ocsp-stapling-verify
Accepted values:
yes, no
Default:
no
Available in AxoSyslog 4.0 and later.
Description: When OCSP stapling verification is enabled, AxoSyslog requests the server to send back its OCSP status. AxoSyslog verifies this status response using the trust store you have configured using the ca-file(), ca-dir(), or the pkcs12-file() options.
Note
RFC 6961 multi-stapling and TLS 1.3-provided multiple responses are currently not validated, only the peer certificate is verified.
IMPORTANT: openssl-conf-cmds() always has the highest priority, so it overrides any other options that can be found in the tls() section.
OpenSSL offers an alternative, software-independent configuration mechanism through the SSL_CONF_cmd interface for setting the various SSL_CTX and SSL options.
The order of operations within openssl-conf-cmds() is significant and the commands are executed in top-down order. This means that if the same option occurs multiple times, then the ’last one wins’. This is also true for options that can be set multiple ways (for example, cipher suites or protocols).
Example configuration:
tls( ca-dir("/etc/ca.d") key-file("/etc/cert.d/serverkey.pem") cert-file("/etc/cert.d/servercert.pem") peer-verify(yes) openssl-conf-cmds(# For system wide available cipher suites use: /usr/bin/openssl ciphers -v# For formatting rules see: https://www.openssl.org/docs/man1.1.1/man3/SSL_CONF_cmd.html"CipherString"=> "ECDHE-RSA-AES128-SHA", # TLSv1.2 and bellow"CipherSuites"=> "TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384", # TLSv1.3+ (OpenSSl 1.1.1+)"Options"=> "PrioritizeChaCha",
"Protocol"=> "-ALL,TLSv1.3",
))
peer-verify()
Accepted values:
optional-trusted
Default:
required-trusted
Description: Verification method of the peer, the four possible values is a combination of two properties of validation:
Whether the peer is required to provide a certificate (required or optional prefix).
Whether the certificate provided needs to be valid or not.
The following table summarizes the possible options and their results depending on the certificate of the peer.
The remote peer has:
no certificate
invalid certificate
valid certificate
Local peer-verify() setting
optional-untrusted
TLS-encryption
TLS-encryption
TLS-encryption
optional-trusted
TLS-encryption
rejected connection
TLS-encryption
required-untrusted
rejected connection
TLS-encryption
TLS-encryption
required-trusted
rejected connection
rejected connection
TLS-encryption
For untrusted certificates only the existence of the certificate is checked, but it does not have to be valid — AxoSyslog accepts the certificate even if it is expired, signed by an unknown CA, or its CN and the name of the machine mismatches.
Warning
When validating a certificate, the entire certificate chain must be valid, including the CA certificate. If any certificate of the chain is invalid, AxoSyslog will reject the connection.
Starting with AxoSyslog version 3.10, you can also use a simplified configuration method for the peer-verify option, simply setting it to yes or no. The following table summarizes the possible options and their results depending on the certificate of the peer.
The remote peer has:
no certificate
invalid certificate
valid certificate
Local peer-verify() setting
no (optional-untrusted)
TLS-encryption
TLS-encryption
TLS-encryption
yes (required-trusted)
rejected connection
rejected connection
TLS-encryption
pkcs12-file()
Accepted values:
Filename
Default:
none
Description: The name of a PKCS #12 file that contains an unencrypted private key, an X.509 certificate, and an optional set of trusted CA certificates.
If this option is used in the configuration, the value of key-file() and cert-file() will be omitted.
You can use the ca-dir() option together with pkcs12-file(). However, this is optional because the PKCS #12 file may contain CA certificates as well.
Passphrase is currently not supported.
Example: Using pkcs12-file()
In the following example, the first command creates a single PKCS #12 file from the private key, X.509 certificate, and CA certificate files. Then, the second half of the example uses the same PKCS #12 file in the AxoSyslog configuration.
Description: A colon-separated list that specifies the supported signature algorithms (in order of decreasing preference) for TLSv1.2 and higher, for example, RSA-PSS+SHA256:ed25519. If this option is not set then all supported signature algorithms supported are permissible.
For servers, it is used to determine which signature algorithms to support.
For clients, this value is used directly for the supported signature algorithms extension.
sni()
Accepted values:
`yes
Default:
no
Description: When set to yes in a destination that uses TLS encryption, this option enables Server Name Indication (also called Server Name Identification, SNI). The AxoSyslog sends the hostname or the IP address set in the destination to the server during the TLS handshake.
Available in AxoSyslog 3.24 and newer.
Example: Using Server Name Indication
The following destination sends the hostname of its destination during the TLS handshake.
comma-separated list of the following options: no-sslv2, no-sslv3, no-tlsv1, no-tlsv11, no-tlsv12, no-tlsv13, none, ignore-hostname-mismatch, ignore-validity-period
Default:
no-sslv2
Available in AxoSyslog 3.7 and newer.
Description: Sets the specified options of the SSL/TLS protocols. You can use it to disable specific protocol versions, and set other options. Note that disabling a newer protocol version (for example, TLSv1.1) does not automatically disable older versions of the same protocol (for example, TLSv1.0). For example, use the following option to permit using only TLSv1.1 or newer:
ssl-options(no-sslv2, no-sslv3, no-tlsv1)
Using ssl-options(none) means that AxoSyslog does not specify any restrictions on the protocol used. However, in this case, the underlying OpenSSL library can restrict the available protocols, for example, certain OpenSSL versions automatically disable SSLv2.
By specifying ignore-hostname-mismatch, you can ignore the subject name of a certificate during the validation process. This means that AxoSyslog checks only that the certificate itself is trusted by the current set of trust anchors (e.g. trusted CAs), and ignores the mismatch between the targeted hostname and the certificate subject. ignore-hostname-mismatch is available in AxoSyslog 4.4 and newer.
By specifying ignore-validity-period, you can ignore the you can ignore the validity periods of certificates during the certificate validation process. ignore-validity-period is available in AxoSyslog 4.5 and newer.
Example: Using ssl-options
The following destination explicitly disables SSL and TLSv1.0
Description: Specifies the permitted SSL/TLS version. Possible values: sslv2, sslv3, tlsv1, tlsv1_0, tlsv1_1, tlsv1_2, tlsv1_3.
trusted-dn()
Accepted values:
list of accepted distinguished names
Default:
none
Description: To accept connections only from hosts using certain certificates signed by the trusted CAs, list the distinguished names of the accepted certificates in this parameter. For example, using trusted-dn("\*, O=Example Inc, ST=Some-State, C=\*") will accept only certificates issued for the Example Inc organization in Some-State state.
trusted-keys()
Accepted values:
list of accepted SHA-1 fingerprints
Default:
none
Description: To accept connections only from hosts using certain certificates having specific SHA-1 fingerprints, list the fingerprints of the accepted certificates in this parameter. for example, trusted-keys("SHA1:00:EF:ED:A4:CE:00:D1:14:A4:AB:43:00:EF:00:91:85:FF:89:28:8F", "SHA1:0C:42:00:3E:B2:60:36:64:00:E2:83:F0:80:46:AD:00:A8:9D:00:15").
To find the fingerprint of a certificate, you can use the following command: openssl x509 -in <certificate-filename>sha1 -noout -fingerprint
Note
When using the trusted-keys() and trusted-dn() parameters, note the following:
First, the trusted-keys() parameter is checked. If the fingerprint of the peer is listed, the certificate validation is performed.
If the fingerprint of the peer is not listed in the trusted-keys() parameter, the trusted-dn() parameter is checked. If the DN of the peer is not listed in the trusted-dn() parameter, the authentication of the peer fails and the connection is closed.
12 - template and rewrite: Format, modify, and manipulate log messages
This chapter explains the methods that you can use to customize, reformat, and modify log messages using AxoSyslog.
Modifying messages using rewrite rules describes how to use macros and templates to format log messages or change the names of logfiles and database tables.
Macros of AxoSyslog lists the different types of macros available in AxoSyslog.
12.1.1 - Formatting messages, filenames, directories, and tablenames
The AxoSyslog application can dynamically create filenames, directories, or names of database tables using macros that help you organize your log messages. Macros refer to a property or a part of the log message, for example, the ${HOST} macro refers to the name or IP address of the client that sent the log message, while ${DAY} is the day of the month when the message was received. Using these macros in the path of the destination log files allows you for example, to collect the logs of every host into separate files for every day.
A set of macros can be defined as a template object and used in multiple destinations.
Another use of macros and templates is to customize the format of the syslog message, for example, to add elements of the message header to the message text.
Note
If a message uses the IETF-syslog format (RFC5424), only the text of the message can be customized (that is, the $MESSAGE part of the log), the structure of the header is fixed.
The AxoSyslog application allows you to define message templates, and reference them from every object that can use a template. Templates can include strings, macros (for example, date, the hostname, and so on), and template functions. For example, you can use templates to create standard message formats or filenames. For a list of macros available in AxoSyslog, see Macros of AxoSyslog. Fields from the structured data (SD) part of messages using the new IETF-syslog standard can also be used as macros.
Template objects have a single option called template-escape(), which is disabled by default (template-escape(no)). This behavior is useful when the messages are passed to an application that cannot handle escaped characters properly. Enabling template escaping (template-escape(yes)) causes AxoSyslog to escape the ', ", and backslash characters from the messages.
Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.
If you do not want to enable the template-escape() option (which is rarely needed), you can define the template without the enclosing braces.
template <template-name> "<template-expression>";
You can also refer to an existing template from within a template. The result of the referred template will be pasted into the second template.
template first-template "sample-text"; template second-template "The result of the first-template is: $(template first-template)";
If you want to use a template only once, you can define the template inline, for example:
Macros can be included by prefixing the macro name with a $ sign, just like in Bourne compatible shells. Although using braces around macro names is not mandatory, and the "$MESSAGE" and "${MESSAGE}" formats are equivalent, using the "${MESSAGE}" format is recommended for clarity.
Macro names are case-sensitive, that is, "$message" and "$MESSAGE" are not the same.
To use a literal $ character in a template, you have to escape it. In AxoSyslog versions 3.4 and earlier, use a backslash \$. In version 3.5 and later, use $$.
Note
To use a literal @ character in a template, use @@.
Default values for macros can also be specified by appending the :- characters and the default value of the macro. If a message does not contain the field referred to by the macro, or it is empty, the default value will be used when expanding the macro. For example, if a message does not contain a hostname, the following macro can specify a default hostname.
${HOST:-default_hostname}
By default, AxoSyslog sends messages using the following template: ${ISODATE} ${HOST} ${MSGHDR}${MESSAGE}\\n. (The ${MSGHDR}${MESSAGE} part is written together because the ${MSGHDR} macro includes a trailing whitespace.)
Example: Using templates and macros
The following template (t_demo_filetemplate) adds the date of the message and the name of the host sending the message to the beginning of the message text. The template is then used in a file destination: messages sent to this destination (d_file) will use the message format defined in the template.
If you do not want to enable the template-escape() option (which is rarely needed), you can define the template without the enclosing braces. The following two templates are equivalent.
Macros can be used to format messages, and also in the name of destination files or database tables. However, they cannot be used in sources as wildcards, for example, to read messages from files or directories that include a date in their name.
12.1.3 - Date-related macros
The macros related to the date of the message (for example: ${ISODATE}, ${HOUR}, and so on) have three further variants each:
S_ prefix, for example, ${S_DATE}: The ${S_DATE} macro represents the date found in the log message, that is, when the message was sent by the original application.
Warning
To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).
R_ prefix, for example, ${R_DATE}: ${R_DATE} is the date when AxoSyslog has received the message.
C_ prefix, for example, ${C_DATE}: ${C_DATE} is the current date, that is when AxoSyslog processes the message and resolves the macro.
The ${DATE} macro equals the ${S_DATE} macro.
The values of the date-related macros are calculated using the original timezone information of the message. To convert it to a different timezone, use the time-zone() option. You can set the time-zone() option as a global option, or per destination. For sources, it applies only if the original message does not contain timezone information. Alternatively, you can modify the timezone of the message using timezone-specific rewrite rules. For details, see Rewrite the timezone of a message.
Converting the timezone changes the values of the following date-related macros (macros MSEC and USEC are not changed):
AMPM
DATE
DAY
FULLDATE
HOUR
HOUR12
ISODATE
ISOWEEK
MIN
MONTH
MONTH_ABBREV
MONTH_NAME
MONTH_WEEK
SEC
STAMP
TZ
TZOFFSET
UNIXTIME
WEEK
WEEK_DAY
WEEK_DAY_ABBREV
WEEK_DAY_NAME
YEAR
YEAR_DAY
12.1.4 - Hard versus soft macros
Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. Hard macros are read-only. Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the pattern database or a CSV-parser. In contrast to hard macros, soft macros are writable and can be modified within AxoSyslog, for example, using rewrite rules.
Hard and soft macros are rather similar and often treated as equivalent. Macros are most commonly used in filters and templates, which does not modify the value of the macro, so both soft and hard macros can be used. However, it is not possible to change the values of hard macros in rewrite rules or via any other means.
The following macros in AxoSyslog are hard macros and cannot be modified: BSDTAG, CONTEXT_ID, DATE, DAY, FACILITY_NUM, FACILITY, FULLDATE, HOUR, IP-PROTO, ISODATE, ISOWEEK, LEVEL_NUM, LEVEL, MIN, MONTH_ABBREV, MONTH_NAME, MONTH, MONTH_WEEK, PRIORITY, PRI, RCPTID, SDATA, SEC, SEQNUM, SOURCEIP, STAMP, TAG, TAGS, TRANSPORT, TZOFFSET, TZ, UNIXTIME, WEEK_DAY_ABBREV, WEEK_DAY_NAME, WEEK_DAY, WEEK, YEAR_DAY, YEAR.
The following macros can be modified:FULLHOST_FROM, FULLHOST, HOST_FROM, HOST, LEGACY_MSGHDR, MESSAGE, MSG,MSGID, MSGONLY, PID, PROGRAM, SOURCE. Custom values created using rewrite rules or parsers can be modified as well, just like stored matches of regular expressions ($0 … $255).
Note that you can modify the timezone of the message, and change the timezone-related macros that way. For details, see Rewrite the timezone of a message.
12.1.5 - Macros of AxoSyslog
The following macros are available in AxoSyslog templates.
Warning
These macros are available when AxoSyslog successfully parses the incoming message as a syslog message, or you use some other parsing method and map the parsed values to these macros.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
AMPM
Description: Typically used together with the ${HOUR12} macro, ${AMPM} returns the period of the day: AM for hours before mid day and PM for hours after mid day. In reference to a 24-hour clock format, AM is between 00:00-12:00 and PM is between 12:00-24:00. 12AM is midnight. Available in AxoSyslog 3.4 and later.
BSDTAG
Description: Facility/priority information in the format used by the FreeBSD syslogd: a priority number followed by a letter that indicates the facility. The priority number can range from 0 to 7. The facility letter can range from A to Y, where A corresponds to facility number zero (LOG_KERN), B corresponds to facility 1 (LOG_USER), and so on.
Custom macros
Description: CSV parsers and pattern databases can also define macros from the content of the messages, for example, a pattern database rule can extract the username from a login message and create a macro that references the username. For details on using custom macros created with CSV parsers and pattern databases, see parser: Parse and segment structured messages and Using parser results in filters and templates, respectively.
DATE, C_DATE, R_DATE, S_DATE
Description: Date of the message using the BSD-syslog style timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example: Jun 13 15:58:00.
DAY, C_DAY, R_DAY, S_DAY
Description: The day the message was sent.
DESTIP
Description: When used, the output specifies the local IP address of the source from which the message originates.
Description: The name of the facility (for example, kern) that sent the message.
FACILITY_NUM
Description: The numerical code of the facility (for example, 0) that sent the message.
FILE_NAME
Description: Name of the log file (including its path) from where AxoSyslog received the message (only available if AxoSyslog received the message from a file or a wildcard-file source). If you need only the path or the filename, use the dirname and basename template functions.
FULLDATE, C_FULLDATE, R_FULLDATE, S_FULLDATE
Description: A nonstandard format for the date of the message using the same format as ${DATE}, but including the year as well, for example: 2006 Jun 13 15:58:00.
FULLHOST
Description: The name of the source host where the message originates from.
If the message traverses several hosts and the chain-hostnames() option is on, the first host in the chain is used.
If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $FULLHOST macro will be the DNS hostname of the host that sent the message to AxoSyslog (that is, the DNS hostname of the last hop). In this case the $FULLHOST and $FULLHOST_FROM macros will have the same value.
If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $FULLHOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.
Note
The use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
Description: The FQDN of the host that sent the message to AxoSyslog as resolved by AxoSyslog using DNS. If the message traverses several hosts, this is the last host in the chain.
The AxoSyslog application uses the following procedure to determine the value of the $FULLHOST_FROM macro:
The AxoSyslog application takes the IP address of the host sending the message.
If the use-dns() option is enabled, AxoSyslog attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $FULLHOST_FROM macro. This value will be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn() is disabled.
If the use-dns() option is disabled, or the address resolution fails, the ${FULLHOST_FROM} macro will return the IP address of the sender host.
Description: The hour of day the message was sent.
HOUR12, C_HOUR12, R_HOUR12, S_HOUR12
Description: The hour of day the message was sent in 12-hour clock format. See also the ${AMPM} macro. 12AM is midnight. Available in AxoSyslog 3.4 and later.
HOST
Description: The name of the source host where the message originates from.
If the message traverses several hosts and the chain-hostnames() option is on, the first host in the chain is used.
If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $HOST macro will be the DNS hostname of the host that sent the message to AxoSyslog (that is, the DNS hostname of the last hop). In this case the $HOST and $HOST_FROM macros will have the same value.
If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $HOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.
Note
The use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.
Description: The FQDN of the host that sent the message to AxoSyslog as resolved by AxoSyslog using DNS. If the message traverses several hosts, this is the last host in the chain.
The AxoSyslog application uses the following procedure to determine the value of the $HOST_FROM macro:
The AxoSyslog application takes the IP address of the host sending the message.
If the use-dns() option is enabled, AxoSyslog attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $HOST_FROM macro. This value will be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn() is disabled.
If the use-dns() option is disabled, or the address resolution fails, the ${HOST_FROM} macro will return the IP address of the sender host.
The IP protocol version used to retrieve or receive the message. Contains either “4” to indicate IPv4 and “6” to indicate IPv6.
ISODATE, C_ISODATE, R_ISODATE, S_ISODATE
Description: Date of the message in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00. If possible, it is recommended to use ${ISODATE} for timestamping. Note that AxoSyslog can produce fractions of a second (for example, milliseconds) in the timestamp by using the frac-digits() global or per-destination option.
Note
As AxoSyslog is precise up to the microsecond, when the frac-digits() option is set to a value higher than 6, AxoSyslog will truncate the fraction seconds in the timestamps after 6 digits.
ISOWEEK, C_ISOWEEK, R_ISOWEEK, S_ISOWEEK
Description: The number of week according to the ISO 8601 standard. Note that the ${WEEK} macro that has been available in returns a non-standard week number that can differ from the value returned by the ${ISOWEEK} macro.
Available in 3.24 and later.
LEVEL_NUM
Description: The priority (also called severity) of the message, represented as a numeric value, for example, 3. For the textual representation of this value, use the ${LEVEL} macro. See PRIORITY or LEVEL for details.
LOGHOST
Description: The hostname of the computer running AxoSyslog.
In version 3.24 and later: the ${LOGHOST} macro returns the fully-qualified domain name (FQDN) only if the use-fqdn() option is set to yes, and the hostname otherwise.
In earlier versions: the ${LOGHOST} macro returns the fully-qualified domain name (FQDN).
MESSAGE
Description: Text contents of the log message without the program name and pid. The program name and the pid together are available in the ${MSGHDR} macro, and separately in the ${PROGRAM} and ${PID} macros.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
The ${MSG} macro is an alias of the ${MESSAGE} macro: using ${MSG} in AxoSyslog is equivalent to ${MESSAGE}.
Note that before AxoSyslog version 3.0, the ${MESSAGE} macro included the program name and the pid. In AxoSyslog 3.0, the ${MESSAGE} macro became equivalent with the ${MSGONLY} macro.
MIN, C_MIN, R_MIN, S_MIN
Description: The minute the message was sent.
MONTH, C_MONTH, R_MONTH, S_MONTH
Description: The month the message was sent as a decimal value, prefixed with a zero if smaller than 10.
Description: The number of the week in the given month (0-5). The week with numerical value 1 is the first week containing a Monday. The days of month before the first Monday are considered week 0. For example, if a 31-day month begins on a Sunday, then the 1st of the month is week 0, and the end of the month (the 30th and 31st) is week 5.
MSEC, C_MSEC, R_MSEC, S_MSEC
Description: The millisecond the message was sent.
Available in AxoSyslog version 3.4 and later.
MQTT_TOPIC
Description: The mqtt() source automatically sets the ${MQTT_TOPIC} name-value pair for the messages it receives. This is useful when the name of the topic contains MQTT wildcards ($, +, #).
Available in AxoSyslog version 4.7 and later.
MSG
The ${MSG} macro is an alias of the ${MESSAGE} macro, using ${MSG} in AxoSyslog is equivalent to ${MESSAGE}. For details on this macro, see MESSAGE.
MSGHDR
Description: The name and the PID of the program that sent the log message in PROGRAM[PID]: format. Includes a trailing whitespace. Note that the macro returns an empty value if both the PROGRAM and PID fields of the message are empty.
MSGID
Description: A string specifying the type of the message in IETF-syslog (RFC5424-formatted) messages. For example, a firewall might use the ${MSGID} “TCPIN” for incoming TCP traffic and the ${MSGID} “TCPOUT” for outgoing TCP traffic. By default, AxoSyslog does not specify this value, but uses a dash (-) character instead. If an incoming message includes the ${MSGID} value, it is retained and relayed without modification.
MSGONLY
Description: Message contents without the program name or pid. Starting with AxoSyslog 3.0, the following macros are equivalent: ${MSGONLY}, ${MSG}, ${MESSAGE}. For consistency, use the ${MESSAGE} macro. For details, see MESSAGE.
PID
Description: The PID of the program sending the message.
PRI
Description: The priority and facility encoded as a 2 or 3 digit decimal number as it is present in syslog messages.
PRIORITY or LEVEL
Description: The priority (also called severity) of the message, for example, error. For the textual representation of this value, use the ${LEVEL} macro. See PRIORITY or LEVEL for details.
PROGRAM
Description: The name of the program sending the message. Note that the content of the ${PROGRAM} variable may not be completely trusted as it is provided by the client program that constructed the message.
PROTO
Description: When used, the output specifies the protocol used on the source from which the message originates.
Description: The original message as received from the client. Note that this macro is available only in 3.16 and later, and only if AxoSyslog received the message using the default-network-drivers-ng() source, or the source receiving the message has the store-raw-message flag set.
RAWMSG_SIZE
Available in AxoSyslog version 4.2 and newer.
Description: The original size of the incoming message in bytes. Might not be available for every source driver.
RCPTID
Description: When the use-rcptid global option is set to yes, AxoSyslog automatically assigns a unique reception ID to every received message. You can access this ID and use it in templates via the ${RCPTID} macro. The reception ID is a monotonously increasing 48-bit integer number, that can never be zero (if the counter overflows, it restarts with 1).
RUNID
Description: An ID that changes its value every time AxoSyslog is restarted, but not when reloaded.
SDATA, .SDATA.SDID.SDNAME
Description: The AxoSyslog application automatically parses the STRUCTURED-DATA part of IETF-syslog messages, which can be referenced in macros. The ${SDATA} macro references the entire STRUCTURED-DATA part of the message, while structured data elements can be referenced using the ${.SDATA.SDID.SDNAME} macro.
Note
When using STRUCTURED-DATA macros, consider the following:
When referencing an element of the structured data, the macro must begin with the dot (.) character. For example, ${.SDATA.timeQuality.isSynced}.
The SDID and SDNAME parts of the macro names are case sensitive: ${.SDATA.timeQuality.isSynced} is not the same as ${.SDATA.TIMEQUALITY.ISSYNCED}.
Example: Using SDATA macros
For example, if a log message contains the following structured data: [exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"] you can use macros like: ${[email protected]} — this would return the Application string in this case.
SEC, C_SEC, R_SEC, S_SEC
Description: The second the message was sent.
SEQNUM
Description: The ${SEQNUM} macro contains a sequence number for the log message. The value of the macro depends on the scenario, and can be one of the following:
If AxoSyslog receives a message via the IETF-syslog protocol that includes a sequence ID, this ID is automatically available in the ${SEQNUM} macro.
If the message is a Cisco IOS log message using the extended timestamp format, then AxoSyslog stores the sequence number from the message in this macro. If you forward this message the IETF-syslog protocol, AxoSyslog includes the sequence number received from the Cisco device in the ${.SDATA.meta.sequenceId} part of the message.
Note
To enable sequence numbering of log messages on Cisco devices, use the following command on the device (available in IOS 10.0 and later): service sequence-numbers. For details, see the manual of your Cisco device.
For locally generated messages (that is, for messages that are received from a local source, and not from the network), AxoSyslog calculates a sequence number when sending the message to a destination (it is not calculated for relayed messages).
The sequence number is not global, but per-destination. Essentially, it counts the number of messages sent to the destination.
This sequence number increases by one for every message sent to the destination. It not lost when AxoSyslog is reloaded, but it is reset when AxoSyslog is restarted.
This sequence number is added to every message that uses the IETF-syslog protocol (${.SDATA.meta.sequenceId}), and can be added to BSD-syslog messages using the ${SEQNUM} macro.
Note
If you need a sequence number for every log message that AxoSyslog receives, use the RCPTID macro.
SOURCE
Description: The identifier of the source statement in the AxoSyslog configuration file that received the message. For example, if AxoSyslog received the log message from the source s_local { internal(); }; source statement, the value of the ${SOURCE} macro is s_local. This macro is mainly useful for debugging and troubleshooting purposes.
SOURCEIP
Description: IP address of the host that sent the message to syslog-ng. (That is, the IP address of the host in the ${FULLHOST_FROM} macro.) Please note that when a message traverses several relays, this macro contains the IP of the last relay.
STAMP, R_STAMP, S_STAMP
Description: A timestamp formatted according to the ts-format() global or per-destination option.
SYSUPTIME
Description: The time elapsed since the AxoSyslog instance was started (that is, the uptime of the AxoSyslog process). The value of this macro is an integer containing the time in 1/100th of the second.
Available in AxoSyslog version 3.4 and later.
TAG
Description: The priority and facility encoded as a 2 digit hexadecimal number.
TAGS
Description: A comma-separated list of the tags assigned to the message.
Note
Note that the tags are not part of the log message and are not automatically transferred from a client to the server. For example, if a client uses a pattern database to tag the messages, the tags are not transferred to the server. A way of transferring the tags is to explicitly add them to the log messages using a template and the ${TAGS} macro, or to add them to the structured metadata part of messages when using the IETF-syslog message format.
When sent as structured metadata, it is possible to reference to the list of tags on the central server, and for example, to add them to a database column.
TRANSPORT
Available in AxoSyslog version 4.5 and later.
AxoSyslog automatically populates this name-value pair with the “transport” mechanism used to retrieve or receive the message. The exact value depends on the source driver that received the message. Currently the following values are implemented:
BSD syslog drivers tcp(), udp() & network()
rfc3164+tls
rfc3164+tcp
rfc3164+udp
rfc3164+proxied-tls
rfc3164+<custom logproto like altp>
UNIX domain drivers unix-dgram(), unix-stream()
unix-stream
unix-dgram
RFC5424-style syslog syslog():
rfc5426: syslog over udp
rfc5425: syslog over tls
rfc6587: syslog over tcp
rfc5424+<custom logproto like altp>: syslog over a logproto plugin
Description: The time-zone as hour offset from GMT, for example: -07:00. In version 1.6.x this used to be -0700 but as ${ISODATE} requires the colon it was added to ${TZOFFSET} as well.
UNIXTIME, C_UNIXTIME, R_UNIXTIME, S_UNIXTIME
Description: Standard UNIX timestamp, represented as the number of seconds since 1970-01-01T00:00:00.
.tls.x509
Description: When using a transport that uses TLS, these macros contain information about the peer’s certificate. That way, you can use information from the client certificate in filenames, database values, or as other metadata. If you clients have their own certificates, then these values are unique per client, but unchangeable by the client. The following macros are available in AxoSyslog version 3.9 and later.
.tls.x509_cn: The Common Name of the certificate.
.tls.x509_o: The value of the Organization field.
.tls.x509_ou: The value of the Organization Unit field.
UNIQID
Description: A globally unique ID generated from the HOSTID and the RCPTID in the format of HOSTID@RCPTID. For details, see use-uniqid() and RCPTID.
Available in AxoSyslog version 3.7 and later.
USEC, C_USEC, R_USEC, S_USEC
Description: The microsecond the message was sent.
Available in AxoSyslog version 3.4 and later.
YEAR, C_YEAR, R_YEAR, S_YEAR
Description: The year the message was sent.
WEEK, C_WEEK, R_WEEK, S_WEEK
Description: The week number of the year, prefixed with a zero for the first nine weeks of the year. (The first Monday in the year marks the first week.)
12.1.5.1 - Example use case: using the $DESTIP, the $DESTPORT, and the $PROTO macros
This section describes scenarios when Axoflow recommends using the $DESTIP, the $DESTPORT, and the $PROTO macros.
Using the $DESTIP, the $DESTPORT, and the $PROTO macros is relevant when multiple sources are configured to receive messages on the AxoSyslog side. In this case, the hostname and IP address on the sender’s side and the AxoSyslog side is the same, and at a later point in the pipeline, AxoSyslog can not by default specify which source received the message. The $DESTIP, the $DESTPORT, and the $PROTO macros solve this issue by specifying the local IP address and local port of the original message source, and the protocol used on the original message source on the AxoSyslog side.
When to use the $DESTIP, the $DESTPORT, and the $PROTO macros
Axoflow recommends using the $DESTIP, the $DESTPORT, and the $PROTO macros in either of the following scenarios:
Your appliance sends out log messages through both UDP and TCP.
The format of the UDP log messages and the TCP log messages is different, and instead of using complex filters, you want to capture either of them, preferably with the simplest possible filter.
The IP addresses on the sender’s side and the AxoSyslog side are the same, therefore the netmask() option doesn’t work in your configuration.
The hostnames on the sender’s side and the AxoSyslog side are the same, therefore the host() option doesn’t work in your configuration.
Macros: $DESTIP, $DESTPORT, and $PROTO
To solve either of the challenges listed previously, AxoSyslog supports the following macros that you can include in your configuration:
With these configuration settings, the macros will specify the local IP, the local port, and the protocol information of the source from which the message originates as follows:
destip=10.12.15.215 destport=5555proto=17
12.1.6 - Using template functions
A template function is a transformation: it modifies the way macros or name-value pairs are expanded. Template functions can be used in template definitions, or when macros are used in the configuration of AxoSyslog. Template functions use the following syntax:
For example, the $(echo) template function simply returns the value of the macro it receives as a parameter, thus $(echo ${HOST}) is equivalent to ${HOST}.
The parameters of template functions are separated by a whitespace character. A template function can have maximum 64 parameters. If you want to use a longer string or multiple macros as a single parameter, enclose the parameter in double-quotes or apostrophes. For example:
$(echo"${HOST}${PROGRAM}${PID}")
Template functions can be nested into each other, so the parameter of a template function can be another template function, like:
$(echo$(echo${HOST}))
For details on the available template functions, see the descriptions of the individual template functions in Template functions of AxoSyslog.
You can define your own template function as a regular configuration object (for example, to reuse the same function in different places in your configuration).
The following template functions are available in AxoSyslog.
base64-encode
Syntax:
$(base64-encode argument)
Description: You can use the base64-encode template function to base64-encode strings and macros. The template function can receive multiple parameters (maximum 64). In this case, AxoSyslog joins the parameters into a single string and encodes this string. For example, $(base64-encode string1 string2) is equivalent to $(base64-encode string1string2).
Available in AxoSyslog version 3.18 and later.
basename
Syntax:
$(basename argument)
Description: Returns the filename from an argument (for example, a macro: $(basename ${FILE_NAME})) that contains a filename with a path. For example, $(basename "/var/log/messages.log") returns messages.log. To extract the path, use the dirname template function.
Available in AxoSyslog version 3.10 and later.
ceil
Syntax:
$(ceil argument)
Description: Rounds a floating-point number upwards to the nearest integer. For example, $(ceil 1.5) is 2, $(ceil -1.5) is -1. See also the floor and round template functions.
Description: The context-lookup template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching messages (for example, the ${MESSAGE}) as a list. It works similarly to the $(grep) template function, but it escapes its output properly, so that the returned value is a list that can be processed with other template functions that work on lists, for example, $(list-slice).
Example: Using the context-lookup template function
The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the tags name-value pair.
$(context-lookup ("${username}"=="root")${tags})
To limit the number of matches that the template function returns, use the --max-count option, for example, $(context-lookup --max-count 5 ("${username}" == "root") ${tags}). If you do not want to limit the number of matches, use --max-count 0.
You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of context-lookup, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.
Available in AxoSyslog version 3.10 and later.
context-values
Syntax:
$(context-values $name-value1 $name-value2 ...)
Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma-separated list of the ${HOST} values that appear in the context.
Available in AxoSyslog version 3.10 and later.
dirname
Syntax:
$(dirname argument)
Description: Returns the path (without the filename) from an argument (for example, a macro: $(basename ${FILE_NAME}) that contains a filename with a path. For example, $(dirname "/var/log/messages.log") returns /var/log path. To extract the filename, use the basename template function.
Available in AxoSyslog version 3.10 and later.
echo
Syntax:
$(echo argument)
Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.
env
Syntax:
$(env <environment-variable>)
Description: Returns the value of the specified environment variable. Available in AxoSyslog 3.5 and later.
explode
Syntax:
$(explode <separator> <string1> <string2> ...)
Description: Turns a string separated by a specific character into a list. You can also use the implode template function, which turns a list into a string combining the pieces together with a separator. Available in AxoSyslog 3.21 and later.
Example: Using the explode template function
The following configuration example turns strings into a list. If there are several strings, AxoSyslog looks for a separator within each individual string. For example, string 2 is separated as string, 2 in the example below:
Configuration
Result
$(explode ';' string1;string 2;string3;string4)
"string1,string,2,string3,string4"
Enclose the strings in double-quotes or apostrophes and string 2 is separated as shown below:
Description: Runs the filter expression on each element of a given list, and returns only those list elements that meet the requirements of the filter expression. The current value is referred by $_, similarly to the map template function.
Note
You can use macros, logical expressions, and template functions inside the expression.
Available in AxoSyslog version 3.30 and later.
Example: using the filter template function in your configuration
When used in configuration as seen in the example, the filter template function filters even numbers from an input list of 0, 1, 2 and 3:
<filter-expression>: Mandatory parameter. The <filter-expression> parameter can be:
a comparison
a filter
a logical expression built from filters (using and, or, and not)
Note
When using the <filter-expression> parameter, you can refer other template functions, or use macros.
To refer to the variable bound to the current element of the list, use $_ macro.
The following examples illustrate several ways that you can use a single filter, or a logical expression built from several filters.
('1'=='1')('$_' le '1')('$(% $_ 2)' eq '0')('$_' le '1') and ('$(% $_ 2)' eq '0')
list: Mandatory parameter. A AxoSyslog list.
format-cef-extension
AxoSyslog version 3.8 includes a new template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.
You can use the value-pairs that AxoSyslog stores about the log message as CEF fields. Using value-pairs, you can:
select which value-pairs to use as CEF fields,
add custom value-pairs as CEF fields,
rename value-pairs, and so on.
For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-\* template functions is different from the syntax of value-pairs(): these template functions use a syntax similar to command lines.
Using the format-cef-extension template function has the following prerequisites:
Set the on-error global option to drop-property, otherwise if the name of a name-value pair includes an invalid character, AxoSyslog drops the entire message. (Key name in CEF extensions can contain only the A-Z, a-z and 0-9 characters.)
options { on-error("drop-property");};
The log messages must be encoded in UTF-8. Use the encoding() option or the validate-utf8 flag in the message source.
Example: Using the format-cef-extension template function
The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
Description: Formats the message into Splunk Common Information Model (CIM) format. Applications that can receive messages in CIM format include Kibana, logstash, and Splunk. Applications that can be configured to log into CIM format include nflog and the Suricata IDS engine.
To use the format-cim() template function, AxoSyslog must be compiled with JSON support. To see if your AxoSyslog binary was compiled with JSON support, execute the syslog-ng --version command.
Description: The $(format-date) template function takes a timestamp in the DATETIME representation and formats it according to an strftime() format string. The DATETIME representation is a UNIX timestamp formatted as a decimal number, with an optional fractional part, where the seconds and the fraction of seconds are separated by a dot.
If the timestamp argument is missing, the timestamp of the message is used.
Options:
--time-zone <TZstring>: Override timezone of the original timestamp
For example: $(format-date --time-zone PST8PDT %Y-%m-%dT%H:%M:%S 1667500613) corresponds to the following format and sets the timezone to Pacific Standard Time, Daylight Saving: “2022-11-03T11:36:53”
The following is a sample log message in EWMM format.
<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
format-flat-json
Syntax:$(format-flat-json parameters)
Description: The format-flat-json template function is identical to the format-json template function, but nested JSON objects are flattened in the output. If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function.
Example: Flattened JSON output
The following example shows the difference between nested and flattened JSON objects.
The output of $(format-json a.b.c=1) is a nested JSON object (whitespace added for better readability):
{"a": {"b": {"c": "1"}}}
The output of $(format-flat-json a.b.c=1) is a flattened JSON object (whitespace added for better readability):
{"a.b.c": "1"}
For details on formatting log messages into JSON format, see format-json.
format-gelf
Syntax:$(format-gelf)
Description: Available in AxoSyslog 3.13 and later.
You can use the Graylog Extended Log Format (GELF) template together with the graylog2() destination to send syslog messages to Graylog. GELF is the native data format of Graylog.
Example: Using the format-gelf template function
The following configuration example shows how you can use the format-gelf template:
Description: The format-json template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json, you can reformat any incoming message as JSON.
You can use the value-pairs that AxoSyslog stores about the log message as JSON fields. Using value-pairs, you can:
select which value-pairs to use as JSON fields,
add custom value-pairs as JSON fields,
rename value-pairs, and so on.
If the value-pair includes type information format-json can propagate it to the next component of the log path. For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-json is different from the syntax of value-pairs(): format-json uses a syntax similar to command lines.
Note
Prior to version 4.0, AxoSyslog handled every data as strings, and allowed you to convert the strings into other types of data that only certain destinations data formats supported.
Starting with AxoSyslog 4.0, each name-value pair is a (name, type, value) triplet, and several components of AxoSyslog have typing support. For details, see Components that support data types.
Example: Using the format-json template function
The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
In the case of syslog-ng macros starting with a dot (for example, “.SDATA.meta.sequenceID”), format-json replaces the dot with an underscore character (for example, {"_SDATA":{"meta":{"sequenceId":"55555"}}}).
To retain the starting dot, use the --leave-initial-dot flag, for example:
If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function. For details, see format-flat-json.
format-welf
This template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELF format is a comma-separated list of name=value elements. Note that the order of the elements is random. If the value contains whitespace, it is enclosed in double-quotes, for example, name="value". For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.
To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For details on selecting value-pairs, see value-pairs().
Example: Using the format-welf() template function
The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).
This template function is deprecated. Use geoip2 instead.
Syntax:$(geoip <IPv4-address>)
Description: This template function returns the 2-letter country code of any IPv4 address or host. IPv6 addresses are not supported. Currently only the 2-letter codes are supported, and only from the default database. For example, $(geoip $HOST)
Note
This template function is available only if AxoSyslog has been compiled with the --enable-geoip compiling option.
Description: This template function extracts specific fields from the mmdb database using the --field parameter. If you omit this parameter, it returns the 2-letter country code of any IPv4/IPv6 address or host.
Note
This template function is available only if AxoSyslog has been compiled with geoip2 support. To enable it, use the --enable-geoip compiling option.
Starting with version 3.24, AxoSyslog tries to automatically detect the location of the database. If that is successful, the database() option is not mandatory.
getent
Syntax:$(getent)
Description: Available in AxoSyslog 3.13 and later.
You can use the getent template function to look up entries from the Name Service Switch libraries, such as, passwd, services, or protocols.
The following databases are supported:
passwd
Use this database to query data related to a user. Specify the user by either username or user ID. You can query the following data: username, user ID, group ID, GECOS field, home directory, or user shell.
The queried data is optional. When you do not query any data, the default behavior applies, which is as follows: user ID is returned for username, or username is returned for user ID.
Username $(getent passwd testuser) returns user ID 1000.
User ID $(getent passwd 1000) returns username testuser.
group
Use this database to query group-related data. The group can be specified using either group ID or group name. You can query the following data: group name, group ID, and members.
$(getent group adm name)$(getent group adm gid)$(getent group adm members)
The queried data is optional. The default behavior is as follows: group ID is returned for group name, or group name is returned for user ID.
Group name $(getent group adm) returns group ID 4.
Group ID $(getent group 4) returns group name adm.
protocols
Use this database to translate protocol name to protocol ID, or protocol ID to protocol string.
$(getent protocols tcp)$(getent protocols 6)
services
Use this database to translate service name to service ID, or service ID to service name.
$(getent services http)$(getent services 80)
graphite-output
Syntax:$(graphite-output parameters)
Description: Available in AxoSyslog 3.6 and later. This template function converts value-pairs from the incoming message to the Graphite plain text protocol format. It is ideal to use with the messages generated by the monitor-source plugin (currently available in the syslog-ng incubator project).
For details on selecting value-pairs in AxoSyslog and for possibilities to specify which information to convert to Graphite plain text protocol format, see Structuring macros, metadata, and other value-pairs. Note that the syntax of graphite-output is different from the syntax of value-pairs(): graphite-output uses a the command-line syntax used in the format-json template function.
Example: Using the graphite-output template function
The following configuration example shows, how to send value-pairs with names starting with “vmstat.” to Graphite running on localhost, port 2003:
Description: The grep template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching message (for example, the ${MESSAGE} field of the message).
Example: Using the grep template function
The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.
$(grep ("${username}"=="root")${auth_method})
You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of grep, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.
hash
Syntax:$(<method> [opts] $arg1 $arg2 $arg3...)
Options:--length N, -l N
Truncate the hash to the first N characters.
Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.
<method> can be one of md5, md4, sha1, sha256, sha512 and “hash”, which is equivalent to md5. Macros are expected as arguments, and they are concatenated without the use of additional characters.
The md4 <method> is deprecated.
This template function can be used for anonymizing sensitive parts of the log message (for example, username) that were parsed out using PatternDB before storing or forwarding the message. This way, the ability of correlating messages along this value is retained.
Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.
Note
These template functions are available only if AxoSyslog has been compiled with the --enable-ssl compile option and the tfhash module has been loaded.
By default, AxoSyslog loads every available module. For details, see Loading modules.
Example: Using the $(hash) template function
The following example calculates the SHA1 hash of the hostname of the message:
$(sha1 $HOST)
The following example calculates the SHA256 hash of the hostname, using the salted string to salt the result:
$(sha1 $HOST salted)
To use shorter hashes, set the --length:
$(sha1 --length 6$HOST)
To replace the hostname with its hash, use a rewrite rule:
Description: Returns the value of the <true template> parameter if the <condition> is true. If the <condition> is false, the value of <false template> is returned.
Example: Using pattern databases and the if template function
The following example returns violation if the username name-value pair of a message is root, and system otherwise.
$(if("${username}"=="root")"violation""system")
This can be used to set the class of a message in pattern database rules based on the condition.
Since template functions can be embedded into each other, it is possible to use another template function as the template of the first one. For example, the following expression returns root if the username is root, admin if the username is joe, and normal user otherwise.
Description: Turns a list into a string combining the pieces together with a separator. You can also use the explode template function, which turns a string separated by a specific character into a list. Available in AxoSyslog 3.21 and later.
Example: Using the implode template function
The following configuration example shows how you can use the implode template to turn a list into a string:
Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.
Example: Using the indent-multi-line template function
The following example writes multi-line messages into a text file.
Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.
Note
This template function is available only if the convertfuncs module has been loaded.
By default, AxoSyslog loads every available module. For details, see Loading modules.
List manipulation
The list-* template functions allow you to manipulate comma-separated lists. Such lists represent a simple array type in AxoSyslog. Note the following about formatting lists:
Values are separated by commas, for example, "item1","item2","item3". The single-element list is an element without a comma.
You can use shell-like quotation to embed commas, for example, "item1","ite\\,m2","item3".
Empty values are skipped (except if they are quoted)
These template functions return a well-formed list, properly encoding and quoting all elements. If a template function returns a single element, all quotation is decoded and the value contains the literal value.
Starting with AxoSyslog version 3.10, the following list-related template functions are available. Certain functions allow you to reference an element using its number: note that the list index starts with zero, so the index of the first element is 0, the second element is 1, and so on.
Description: Returns a list and appends the values of the specified name-value pairs to the end of the list. You can also append elements to an empty list, for example, $(list-append '' 'element-to-add')
Description: This template function creates (concatenates) a list of the values it receives as parameter. The values can be single values (for example, ${HOST}) or lists.
For example, the value of the $(list-concat ${HOST}, ${PROGRAM}, ${PID}) is a comma-separated list.
You can concatenate existing lists into a single list using:
$(list-concat ${list1}${list2})
list-count
Syntax:$(list-count ${list} )
Description: Returns the number of elements in the list.
list-head
Syntax:$(list-head ${list} )
Description: Returns the first element of the list, unquoted.
list-nth
Syntax:$(list-nth <index-number> ${list} )
Description: Returns the nth element of the list, unquoted. Note that the list index starts with zero, so (list-nth 1 ${list} ) returns the second element, and so on.
list-search
Syntax:$(list-search [OPTIONS] <pattern> ${list})
Description: The list-search template function searches the elements of ${list} starting at the specified start_index, then returns the index of the first match of <pattern> within ${list}`.
Note
Indexing is 0-based. If <pattern> is not found, the function returns an empty string.
Options:
--mode MODE: Matching mode, with the following possible values: literal (default), prefix, substring, glob, pcre
--start-index N: Skips N elements in the ${list}
list-slice
Syntax:$(list-slice <from>:<to> ${list} )
Description: Returns the specified subset of the list. Note that the list index starts with zero, for example, $(list-slice 1:2 ${list} ) returns the second and third element of the list, and so on.
You can omit the from or to index if you want to start the subset from the beginning or end of the list, for example: 3: returns the list starting with the 4th element, while :3 returns the first four elements.
Negative numbers select an element from the end of the list, for example, -3: returns the last three element of the list.
list-tail
Syntax:$(list-tail ${list} )
Description: Returns the list without the first element. For example, if the ${mylist} list contains the one, two, three elements, then $(list-tail ${mylist} ) returns two, three.
length
Syntax:$(length "<macro>")
Description: Returns the length of the macro in characters, for example, the length of the message. For example, the following filter selects messages that are shorter than 16 characters:
f_short { match ('-', value ("$(if($(length "${MESSAGE}") <= 16) "-" "+")"));};
lowercase
Syntax:$(lowercase "<macro>")
Description: Returns the lowercase version of the specified string or macro. For example, the following example uses the lowercase version of the hostname in a directory name:
Description: These template functions allow you to manipulate numbers, that is, to perform addition (+), substraction (-), multiplication (*), division (/), and modulus (%). All of them require two numeric arguments. The result is NaN (Not-a-Number) if the parameters are not numbers, cannot be parsed, or if a division by zero would occur. For example, to add the value of two macros, use the following template function:
$(+ "${<MACRO1>}""${<MACRO2>}");
Starting with AxoSyslog version 3.22 and later, the numerical operators support floating-point values. They behave like the operators in the C programming language:
If both operands are integers, they return an integer.
If any of the operands is a floating-point number, they return a floating-point result.
For example:
$(/ 3 2)# Both operands are integers, so the result is 1# One of the operands is a floating point number, so the result is also floating-point$(/ 3.0 2)# = 1.500000$(/ 3 2.0)# = 1.500000$(/ 3.0 2.0)# = 1.500000
To round floating-point numbers, you can use the ceil, floor, and round template functions.
When you are correlating messages and a name-value pair contains numerical values in the messages, you can calculate the lowest (min), highest (max), total (sum), and mean (average) values. These calculations process every message of the correlation context. For details on message correlation, see Correlating log messages. For example, if the messages of the context have a .myfields.load name-value pair, you can find the highest load value using the following template function.
$(max ${.myfields.load})
or
Syntax:$(or <macro1> <macro2>)
Description: This template function returns the first non-empty argument.
Description: This template function returns the value of its first parameter (a string or macro), prepended with a string. This string is <width> long, and repeats the character or string set in the third parameter. If you use a single character, it is added >times. If you use a string, it is repeated until its length reaches <width>>The default padding character is ’ ’ (space). For example:
Example: Using the padding template function
If the value of the ${MESSAGE} macro is mymessage, then the output of the padding() template function is the following:
Description: This template function enables you to write a custom template function in Python. You can define a Python block in your AxoSyslog configuration file, define one or more Python functions in it, and use the methods as template functions. If you use a Python block, AxoSyslog embeds a Python interpreter to process the messages.
The following points apply to using Python blocks in AxoSyslog in general:
Python parsers and template functions are available in AxoSyslog version 3.10 and later.
Python destinations and sources are available in AxoSyslog version 3.18 and later.
Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).
The Python block must be a top-level block in the AxoSyslog configuration file.
If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.
In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".
The Python object is initiated every time when AxoSyslog is started or reloaded.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
The Python block can contain multiple Python functions.
Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.
Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.
Python error messages are available in the internal() source of AxoSyslog.
You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.
The following points apply to Python parsers.
The first argument in the definition of the Python function is the actual log message. This is implicitly passed to the function, you do not have to use it in the template function.
The value of the template function is return value of the Python function.
To reference a name-value pair or a macro in the Python function, use the dot-notation. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message.HOST, and so on.
You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.
The following example creates a Python template function called resolve_host that receives an IP address as an argument, and attempts to resolve it into a hostname.
Description: Replaces the delimiter character with a new one. For example, the following example replaces the tabulators (\\t) in the message with semicolons (;):
$(replace-delimiter "\t"";""${MESSAGE}")
Available in AxoSyslog 3.5 and later.
round
Syntax:$(round argument)
Description: Rounds a floating-point number to the nearest integer. For example, $(round 1.5) is 2. See also the ceil and floor template functions.
This template function has an optional second argument that sets the precision of rounding. The default is 0 (output a natural number), but values up to 20 are accepted. For example, $(round 2.123456 4) is 2.1235.
Description: This file replaces the special characters in macro values, for example, it can replace the slash (/) characters in a filename with the underscore (_) character. If you specify multiple arguments, they will be concatenated using the / character, so they can be used as separate directory levels when used in filenames.
The function has the following options:
--ctrl-chars or -c
Filter control characters (characters that have an ASCII code of 32 or lower). This option is used by default.
--invalid-chars <characterlist> or -i <characterlist>
The list of characters to be replaced with underscores (_). The default list contains the / character. The following example replaces the \ and @ characters, so for example, fo\o@bar becomes foobar:
$(sanitize -i \@$PROGRAM)
--no-ctrl-chars or -C
Do not filter the control characters (characters that have an ASCII code of 32 or lower).
--replacement <replacement-character> or -r <replacement-character>
The character used to replace invalid characters. By default, this is the underscore (_). The following example replaces invalid characters with colons instead of underscores, so for example, foo/bar becomes foo;bar:
$(sanitize -r ;$PROGRAM)
Example: Using the sanitize template function
The following example uses the sanitize function on two macros, and the results are used as directory names in a file destination.
This is equivalent to file("/var/log/$HOST/$PROGRAM/messages");, but any slashes in the values of the $HOST and $PROGRAM macros are replaced with underscores.
stardate
Syntax:$(stardate [option] "<date-in-unixtime>")
Description: Converts a date in UNIXTIME (for example, ${UNIXTIME}) into stardate, displaying the year and the progress of the year in a number of digits (YYYY.NNN). You can set the number of digits using the --digits option, for example:
$(stardate --digits 2"${R_UNIXTIME}")
strip
Syntax:$(strip "<macro>")
Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:
Description: This function extracts a substring of a string.
argument The string to extract the substring from, for example, "${MESSAGE}"
offset Specifies where the substring begins (in characters). 0 means to start from the beginning of the string, 5 means to skip the first 5 characters of the string, and so on. Use negative numbers to specify where to start from the end of the string, for example, -1 means the last character, -5 means to start five characters before the end of the string.
lengthOptional parameter: The number of characters to extract. If not specified, the substring will be extracted from the offset to the end of the string. Use negative numbers to stop the substring before the end of the string, for example, -5 means the substring ends five characters before the end of the string.
Example: Using the substr template function
Skip the first 15 characters of the message, and select the rest:
$(substr "${MESSAGE}""15");
Select characters 16-30 of the message (15 characters with offset 15):
$(substr "${MESSAGE}""15""15");
Select the last 15 characters of the message:
$(substr "${MESSAGE}""-15");
A template that converts the message to RFC3164 (BSD-syslog) format and truncates the messages to 1023 characters:
If you do not set the value-if-set and value-if-unset arguments, the $(tag) template function acts as a boolean and expands to 0 or 1, depending on whether the message has the specified tag set.
If the value-if-set and value-if-unset arguments are set, $(tag) returns a string: the second argument (<value-if-set>) if the message has <tag>, and the third argument (<value-if-unset>) if the message doesn’t have <tag>.
Description: This template function looks up the in the configuration and uses that to format its result. The referenced template can be static or dynamic. For static templates, AxoSyslog resolves the template when it starts, or when the configuration is reloaded. For dynamic templates, the results are resolved runtime (for dynamic templates, the template name contains at least one ‘$’ character). For example, the name of the template to be invoked can be extracted from the message, or from a name-value pair set using the add-contextual-data() feature.
For dynamic templates, you can set an optional second template. This second template will be the results of the template function if resolving the dynamic template fails for some reason. For example:
Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
Description: You can use the url-decode template function to decode url-encoded strings and macros. For example, $(url-decode %3C%3E) yields <>. The url-decode` can receive multiple parameters (maximum 64). In this case, each parameter is decoded separately, and simply concatenated.
Available in AxoSyslog version 3.18 and later.
url-encode
Syntax:$(url-encode ${MESSAGE})
Description: The url-encode template function escapes strings for use as URI path or query parameter segments or form encoded HTTP POST data. All input characters that are not a-z, A-Z, 0-9, ‘-’, ‘.’, ‘_’ or ‘~’ are converted to their “URL escaped” version.
Available in AxoSyslog version 3.18 and later. (In version 3.16-3.17, this template function was called urlencode.)
uuid
Syntax:$(uuid)
Description: Generates a Universally Unique IDentifier (UUID) that complies with RFC4122. That way, an UUID can be added to the message soon after it is received, so messages stored in multiple destinations can be identified. For example, when storing messages in a database and also in files, the UUID can be used to find a particular message both in the database and the files.
To generate a UUID, you can use a rewrite rule to create a new value-pair for the message.
Example: Using Universally Unique Identifiers
The following example adds a value-pair called MESSAGE_UUID to the message using a rewrite rule and a template.
This template function is available only if the tfuuid module has been loaded.
By default, AxoSyslog loads every available module. For details, see Loading modules.
12.1.8 - Modifying the on-the-wire message format
Macros, templates, and template functions allow you to fully customize the format of the message. This flexibility makes it possible to use AxoSyslog in some unexpected way if needed, for example, to emulate simple, plain-text protocols. The following example shows you how to send LPUSH commands to a Redis server.
Note
The purpose of this example is to demonstrate the flexibility of AxoSyslog. A dedicated Redis destination is available in AxoSyslog version 3.5. For details, see redis: Store name-value pairs in Redis.
The following template is a valid LPUSH command in accordance with the Redis protocol, and puts the $MESSAGE into a separate list for every $PROGRAM:
The AxoSyslog application can rewrite parts of the messages using rewrite rules. Rewrite rules are global objects similar to parsers and filters and can be used in log paths. The AxoSyslog application has two methods to rewrite parts of the log messages: substituting (setting) a part of the message to a fix value, and a general search-and-replace mode.
Substitution completely replaces a specific part of the message that is referenced using a built-in or user-defined macro.
General rewriting searches for a string in the entire message (or only a part of the message specified by a macro) and replaces it with another string. Optionally, this replacement string can be a template that contains macros.
Rewrite rules are similar to filters: they must be defined in the syslog-ng.conf configuration file and used in the log statement. You can also define the rewrite rule inline in the log path.
Note
The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.
12.2.1 - Replacing message parts
To replace a part of the log message, you have to:
define a string or regular expression to find the text to replace
define a string to replace the original text (macros can be used as well)
select the field of the message that the rewrite rule should process
Substitution rules can operate on any soft macros, for example, MESSAGE, PROGRAM, or any user-defined macros created using parsers. You can also rewrite the structured-data fields of messages complying to the RFC5424 (IETF-syslog) message format.
Note
Hard macros cannot be modified. For details on the hard and soft macros, see Hard versus soft macros).
Substitution rules use the following syntax:
Declaration:
rewrite <name_of_the_rule> { subst("<string or regular expression to find>",
"<replacement string>", value(<field name>), flags());};
The type() and flags() options are optional. The type() specifies the type of regular expression to use, while the flags() are the flags of the regular expressions. For details on regular expressions, see Regular expressions.
A single substitution rule can include multiple substitutions that are applied sequentially to the message. Note that rewriting rules must be included in the log statement to have any effect.
Note
For case-insensitive searches, add the flags(ignore-case) option. To replace every occurrence of the string, add flags(global) option. Note that the store-matches flag is automatically enabled in rewrite rules.
Example: Using substitution rules
The following example replaces the IP in the text of the message with the string IP-Address.
12.2.2 - Setting message fields to specific values
To set a field of the message to a specific value, you have to:
define the string to include in the message, and
select the field where it should be included.
You can set the type of the field. Where you can use of templates in set() and groupset(), you can use type-casting, and the type information is properly promoted. For details, see Specifying data types in value-pairs.
12.2.3 - Setting severity with the set-severity() rewrite function
It is possible to configure the severity field with the set-severity() rewrite function. When configured, the set-severity() rewrite function will only rewrite the $SEVERITY field in the message to the first parameter value specified in the function.
Note
If the parameter value is not a valid parameter value, the function ignores it and sends a debug message, but the AxoSyslog application still sends the message.
Declaration
rewrite <name_of_the_rule> { set-severity("severity string or number");};
Parameters
The set-severity() rewrite function has a single, mandatory parameter that can be defined as follows:
`set-severity("parameter1");`
Accepted values
The set-severity() rewrite function accepts numeric values, named values, and aliases. Aliases are available in AxoSyslog version 4.6 and later.
Numerical Code
Named Value
Alias
0
emerg
SYSLOG_SEVERITY_CODE(0)
0
emergency
SYSLOG_SEVERITY_CODE(0)
0
panic
SYSLOG_SEVERITY_CODE(0)
1
alert
SYSLOG_SEVERITY_CODE(1)
2
crit
SYSLOG_SEVERITY_CODE(2)
2
critical
SYSLOG_SEVERITY_CODE(2)
2
fatal
SYSLOG_SEVERITY_CODE(2)
3
err
SYSLOG_SEVERITY_CODE(3)
3
error
SYSLOG_SEVERITY_CODE(3)
4
warning
SYSLOG_SEVERITY_CODE(4)
4
warn
SYSLOG_SEVERITY_CODE(4)
5
notice
SYSLOG_SEVERITY_CODE(5)
6
info
SYSLOG_SEVERITY_CODE(6)
6
log
SYSLOG_SEVERITY_CODE(6)
7
debug
SYSLOG_SEVERITY_CODE(7)
Example usage for the set-severity() rewrite function
The following examples can be used in production for the set-severity() rewrite function.
Example using string:
rewrite { set-severity("info");};
Example using numeric string:
rewrite { set-severity("6");};
Example using template:
rewrite { set-severity("${.json.severity}");};
12.2.4 - Setting the facility field with the set-facility() rewrite function
It is possible to set the facility field with the set-facility() rewrite function. When set, the set-facility() rewrite function will only rewrite the $PRIORITY field in the message to the first parameter value specified in the function.
Note
If the parameter value is not a valid parameter value, the function ignores it and sends a debug message, but the application still sends the message.
12.2.5 - Setting the priority of a message with the set-pri() rewrite function
You can set the PRI value of a BSD or IETF syslog message with the set-pri() rewrite function by specifying a template string. This is useful, for example, if incoming messages do not have a PRI value specified by default, but a PRI value is required for filtering purposes.
When configured, the set-pri() function will only rewrite the PRI value of the message field.
Note
If the specified parameter value is not a valid value, the function ignores it and sends a debug message. However, the AxoSyslog application will still send the message.
The set-pri() rewrite function expects a template string as its only parameter, for example:
set-pri(“42”);
set-pri("$.json.priority");
Accepted values
The template string specified for the set-pri() rewrite function must expand to a natural number in the interval of 0–1023, inclusive. This means that if you, for example, extract the value from a syslog <PRI> header (such as <42>), then you need to remove the opening and closing brackets (<>) in the specified template string.
Example: Temporarily raising the priority of an application
In the following example, the set-pri() rewrite function is used to temporarily raise the priority of the application myprogram:
As the application message contains a valid priority field, you can use the set-pri() rewrite function to modify the priority of the message:
set-pri("$.json.priority");
12.2.6 - Setting match variables with the set-matches() rewrite rule
Match macros ($1, $2, ... $255) are temporary variables. You can use them for general purposes when operating with list-like items. For example, the match() filter stores capture group results in match variables when the store-matches flag is set, or the JSON parser produces match variables if the parsed JSON data is an array.
It is possible to set match variables in a single operation with the set-matches() rewrite function. set-matches() uses AxoSyslog list expressions to set $1, $2, ... $255, so it can be considered as a conversion function between AxoSyslog lists and match variables.
Note
To convert match variables into a AxoSyslog list, use the $\* macro, which can be further manipulated using list template functions, or turned into a list in type-aware destinations.
Note
To reset match variables to be empty, use the unset-matches() rewrite rule.
Declaration
rewrite <name_of_the_rule> { set-matches("<list-expression or list-based template function>");};
Example usage for the set-matches() rewrite function
In the following two examples, $1, $2, and $3 will be set to foo, bar, and baz, respectively.
If you want to change the name of a field of a message, you can use rename() rewrite rules. This can be also achieved via using set() and unset() but those require extra conditions and two operation instead of one.
The rename() rewrite rule uses positional arguments and they are both required. It supports condition rewrite. For more information, see Conditional rewrites.
If you use RFC5424-formatted (IETF-syslog) messages, you can also create custom fields in the SDATA part of the message (For details on the SDATA message part, see The STRUCTURED-DATA message part). According to RFC5424, the name of the field (its SD-ID) must not contain the @ character for reserved SD-IDs. Custom SDATA fields must be in the following format: .SDATA.group-name@<private enterprise number>.field-name, for example, [email protected]. (18372.4 is the private enterprise number of Axoflow, the developer of AxoSyslog.)
Example: Rewriting custom SDATA fields
The following example sets the sequence ID field of the RFC5424-formatted (IETF-syslog) messages to a fixed value. This field is a predefined SDATA field with a reserved SD-ID, therefore its name does not contain the @ character.
It is also possible to set the value of a field that does not exist yet, and create a new, custom name-value pair that is associated with the message. The following example creates the [email protected] field and sets its value to yes. If you use the ${[email protected]} macro in a template or SQL table, its value will be yes for every message that was processed with this rewrite rule, and empty for every other message.
The next example creates a new SDATA field-group and field called custom and sourceip, respectively:
If you use the ${[email protected]} macro in a template or SQL table, its value will be that of the SOURCEIP macro (as seen on the machine where the SDATA field was created) for every message that was processed with this rewrite rule, and empty for every other message.
You can verify whether or not the format is correct by looking at the actual network traffic. The SDATA field-group will be called [email protected], and sourceip will become a field within that group. If you decide to set up several fields, they will be listed in consecutive order within the field-group’s SDATA block.
12.2.10 - Setting multiple message fields to specific values
The groupset() rewrite rule allows you to modify the value of multiple message fields at once, for example, to change the value of sensitive fields extracted using patterndb, or received in a JSON format. (If you want to modify the names of message fields, see map-value-pairs: Rename value-pairs to normalize logs.)
The first parameter is the new value of the modified fields. This can be a simple string, a macro, or a template (which can include template functions as well).
The second parameter (values()) specifies the fields to modify. You can explicitly list the macros or fields (a space-separated list with the values enclosed in double-quotes), or use wildcards and glob expressions to select multiple fields.
Note that groupset() does not create new fields, it only modifies existing fields.
You can refer to the old value of the field using the $_ macro. This is resolved to the value of the current field, and is available only in groupset() rules.
You can set the type of the field. Where you can use of templates in set() and groupset(), you can use type-casting, and the type information is properly promoted. For details, see Specifying data types in value-pairs.
12.2.11 - map-value-pairs: Rename value-pairs to normalize logs
The map-value-pairs() parser allows you to map existing name-value pairs to a different set of name-value pairs. You can rename them in bulk, making it easy to use for log normalization tasks (for example, when you parse information from different log messages, and want to convert them into a uniform naming scheme). You can use the normal value-pairs expressions, similarly to value-pairs based destinations. Using map-value-pairs() retains type data if available.
The following example creates a new name-value pair called username, adds the hashed value of the .apache.username to this new name-value pair, then adds the webserver prefix to the name of every name-value pair of the message that starts with .apache
Starting with 3.2, it is possible to apply a rewrite rule to a message only if certain conditions are met. The condition() option effectively embeds a filter expression into the rewrite rule: the message is modified only if the message passes the filter. If the condition is not met, the message is passed to the next element of the log path (that is, the element following the rewrite rule in the log statement, for example, the destination). Any filter expression normally used in filters can be used as a rewrite condition. Existing filter statements can be referenced using the filter() function within the condition. For details on filters, see Filters.
Note
Using conditions in rewrite rules can simplify your AxoSyslog configuration file, as you do not need to create separate log paths to modify certain messages.
Using conditional rewrite
The following procedure summarizes how conditional rewrite rules (rewrite rules that have the condition() parameter set) work. The following configuration snippet is used to illustrate the procedure:
The log path receives a message from the source (s1).
The rewrite rule (r_rewrite_set) evaluates the condition. If the message matches the condition (the PROGRAM field of the message is “myapplication”), AxoSyslog rewrites the log message (sets the value of the HOST field to “myhost”), otherwise it is not modified.
The next element of the log path processes the message (d1).
Example: Using conditional rewriting
The following example sets the HOST field of the message to myhost only if the message was sent by the myapplication program.
Assuming the sender is sending messages in near-real-time, AxoSyslog can guess the timezone
By default, these operations modify the date-related macros of the message that correspond to the date the message was sent (that is, the S_ macros). You can modify the dates when AxoSyslog has received the messages (that is, the R_ macros), but this is rarely needed. To do so, include the time-stamp(recvd) option in the operation, for example:
Use the fix-time-zone() operation to correct the timezone of a message if it was parsed incorrectly for some reason, or if the client did not include any timezone information in the message. You can specify the new timezone as the name of a timezone, or as a template string. For example, use the following rewrite rule to set the timezone to EST5EDT:
rewrite { fix-time-zone("EST5EDT");};
If you have lots of clients that do not send timezone information in the log messages, you can create a database file that stores the timezone of the clients, and feed this data to AxoSyslog using the add-contextual-data() feature. For details, see Adding metadata from an external file.
guess-time-zone()
Use the guess-time-zone() operation attempts to set the timezone of the message automatically, using heuristics on the timestamps. Normally the AxoSyslog application performs this operation automatically when it parses the incoming message. Using this operation in a rewrite rule can be useful if you cannot parse the incoming message for some reason (and use the flags(no-parse) option in your source, but you want to set the timezone automatically later (for example, after you have preprocessed the message).
Using this operation is identical to using the flags(guess-timezone) flag in the source.
set-time-zone()
Use the set-time-zone() operation to set the timezone of the message to a specific value, that is to convert an existing timezone to a different one. This operation is identical to setting the time-zone() option in a destination or as a global option, but can be applied selectively to the messages using conditions.
12.2.15 - Anonymizing credit card numbers
Log messages of banking and e-commerce applications might include credit card numbers (Primary Account Number or PAN). According to privacy best practices and the requirements of the Payment Card Industry Data Security Standards (PCI-DSS), PAN must be rendered unreadable. The AxoSyslog application uses a regular expression to detect credit card numbers, and provides two ways to accomplish this: you can either mask the credit card numbers, or replace them with a hash. To mask the credit card numbers, use the credit-card-mask() or the credit-card-hash() rewrite rules in a log path.
By default, these rewrite rules process the MESSAGE part of the log message.
credit-card-hash()
Synopsis:
credit-card-hash(value(""))
Description: Process the specified message field (by default, ${MESSAGE}), and replace any credit card numbers (Primary Account Number or PAN) with a 16-character-long hash. This hash is generated by calculating the SHA-1 hash of the credit card number, selecting the first 64 bits of this hash, and representing this 64 bits in 16 characters.
credit-card-mask()
Synopsis:
credit-card-mask(value(""))
Description: Process the specified message field (by default, ${MESSAGE}), and replace the 7-12th character of any credit card numbers (Primary Account Number or PAN) with asterisks (\*). For example, AxoSyslog replaces the number 5542043004559005 with 554204\*\*\*\*\*\*9005.
12.3 - Regular expressions
Filters and substitution rewrite rules can use regular expressions. In regular expressions, the characters ()[].\*?+^$|\\ are used as special symbols. Depending on how you want to use these characters and which quotation mark you use, these characters must be used differently, as summarized below.
Strings between single quotes ('string') are treated literally and are not interpreted at all, you do not have to escape special characters. For example, the output of '\\x41' is \\x41 (characters as follows: backslash, x(letter), 4(number), 1(number)). This makes writing and reading regular expressions much more simple: it is recommended to use single quotes when writing regular expressions.
When enclosing strings between double-quotes ("string"), the string is interpreted and you have to escape special characters, that is, to precede them with a backslash (\\) character if they are meant literally. For example, the output of the "\\x41" is simply the letter a. Therefore special characters like \\(backslash) or "(quotation mark) must be escaped (\\\\ and \\"). The following expressions are interpreted: \\a, \\n, \\r, \\t, \\v. For example, the \\$40 expression matches the $40 string. Backslashes have to be escaped as well if they are meant literally, for example, the \\\\d expression matches the \\d string.
Note
If you use single quotes, you do not need to escape the backslash, for example, match("\\\\.") is equivalent to match('\\.').
Enclosing alphanumeric strings between double-quotes ("string") is not necessary, you can just omit the double-quotes. for example, when writing filters, match("sometext") and match(sometext) will both match for the sometext string.
Note
Only strings containing alphanumerical characters can be used without quotes or double quotes. If the string contains whitespace or any special characters (()[].\*?+^$|\\ or ;:#), you must use quotes or double quotes.
When using the ;:# characters, you must use quotes or double quotes, but escaping them is not required.
By default, all regular expressions are case sensitive. To disable the case sensitivity of the expression, add the flags(ignore-case) option to the regular expression.
Adding the flags(ignore-case) option to glob patterns does not disable case sensitivity.
The regular expressions can use up to 255 regexp matches (${1} ... ${255}), but only from the last filter and only if the flags("store-matches") flag was set for the filter. For case-insensitive searches, use the flags("ignore-case") option.
12.3.1 - Options of regular expressions
This chapter lists regular expressions supported by AxoSyslog and their available supported type() and flags() options.
By default, AxoSyslog uses PCRE-style regular expressions. To use other expression types, add the type() option after the regular expression.
The AxoSyslog application supports the following regular expression type() options:
12.3.1.1 - The type() options of regular expressions
By default, AxoSyslog uses PCRE-style regular expressions, which are supported on every platform starting with AxoSyslog version 3.1. To use other expression types, add the type() option after the regular expression.
The AxoSyslog application supports the following type() options:
Perl Compatible Regular Expressions (pcre)
Description: Uses Perl Compatible Regular Expressions (PCRE). If the type() parameter is not specified, AxoSyslog uses PCRE regular expressions by default.
Description: Matches the strings literally, without regular expression support. By default, only identical strings are matched. For partial matches, use the flags("prefix") or the flags("substring") flags.
Glob patterns without regular expression support (glob)
Description: Matches the strings against a pattern containing \* and ? wildcards, without regular expression and character range support. The advantage of glob patterns to regular expressions is that globs can be processed much faster.
\*: matches an arbitrary string, including an empty string
?: matches an arbitrary character
The wildcards can match the / character.
You cannot use the \* and ? literally in the pattern.
12.3.1.2 - The flags() options of regular expressions
Similarly to the type() options, the flags() options are also optional within regular expressions.
The following list describes each type() option’s flags() options.
Starting with AxoSyslog version 3.1, PCRE expressions are supported on every platform. If the type() parameter is not specified, AxoSyslog uses PCRE regular expressions by default.
The following example shows the structure of PCRE-style regular expressions in use.
Usable only in rewrite rules, flags("global") matches for every occurrence of the expression, not only the first one.
ignore-case
Disables case-sensitivity.
newline
When configured, it changes the newline definition used in PCRE regular expressions to accept either of the following:
a single carriage-return
linefeed
the sequence carriage-return and linefeed (\\r, \\n and \\r\\n, respectively)
This newline definition is used when the circumflex and dollar patterns (^ and $) are matched against an input. By default, PCRE interprets the linefeed character as indicating the end of a line. It does not affect the \\r, \\n or \\R characters used in patterns.
store-matches
Stores the matches of the regular expression into the $0, ... $255 variables. The $0 stores the entire match, $1 is the first group of the match (parentheses), and so on. Named matches (also called named subpatterns), for example, (?<name>...), are stored as well. Matches from the last filter expression can be referenced in regular expressions.
Note
To convert match variables into a AxoSyslog list, use the $\* macro, which can be further manipulated using List manipulation, or turned into a list in type-aware destinations.
unicode
Uses Unicode support for UTF-8 matches: UTF-8 character sequences are handled as single characters.
utf8
An alias for the unicode flag.
12.3.1.2.2 - Literal string searches
Literal string searches have the following flags() options:
global
Usable only in rewrite rules, flags("global") matches for every occurrence of the expression, not only the first one.
ignore-case
Disables case-sensitivity.
prefix
During the matching process, patterns (also called search expressions) are matched against the input string starting from the beginning of the input string, and the input string is matched only for the maximum character length of the pattern. The initial characters of the pattern and the input string must be identical in the exact same order, and the pattern’s length is definitive for the matching process (that is, if the pattern is longer than the input string, the match will fail).
Example: matching / non-matching patterns for the input string ’exam'
For the input string 'exam',
the following patterns will match:
'ex' (the pattern contains the initial characters of the input string in the exact same order)
'exam' (the pattern is an exact match for the input string)
the following patterns will not match:
'example' (the pattern is longer than the input string)
'hexameter' (the pattern’s initial characters do not match the input string’s characters in the exact same order, and the pattern is longer than the input string)
store-matches
Stores the matches of the regular expression into the $0, ... $255 variables. The $0 stores the entire match, $1 is the first group of the match (parentheses), and so on. Named matches (also called named subpatterns), for example, (?<name>...), are stored as well. Matches from the last filter expression can be referenced in regular expressions.
Note
To convert match variables into a AxoSyslog list, use the $\* macro, which can be further manipulated using List manipulation, or turned into a list in type-aware destinations.
substring
The given literal string will match when the pattern is found within the input. Unlike flags("prefix"), the pattern does not have to be identical with the given literal string.
12.3.1.2.3 - Glob patterns without regular expression support
There are no supported flags() options for glob patterns without regular expression support.
12.3.2 - Optimizing regular expressions
The host(), match(), and program() filter functions and some other objects accept regular expressions as parameters. But evaluating general regular expressions puts a high load on the CPU, which can cause problems when the message traffic is very high. Often the regular expression can be replaced with simple filter functions and logical operators. Using simple filters and logical operators, the same effect can be achieved at a much lower CPU load.
Example: Optimizing regular expressions in filters
Suppose you need a filter that matches the following error message logged by the xntpd NTP daemon:
xntpd[1567]: time error -1159.777379 is too large (set clock manually);
The following filter uses regular expressions and matches every instance and variant of this message.
filter f_demo_regexp { program("demo_program") and
match("time error .* is too large .* set clock manually");};
Segmenting the match() part of this filter into separate match() functions greatly improves the performance of the filter.
filter f_demo_optimized_regexp { program("demo_program") and
match("time error") and
match("is too large") and
match("set clock manually");};
13 - parser: Parse and segment structured messages
The filters and default macros of AxoSyslog work well on the headers and metainformation of the log messages, but are rather limited when processing the content of the messages. Parsers can segment the content of the messages into name-value pairs, and these names can be used as user-defined macros. Subsequent filtering or other type of processing of the message can use these custom macros to refer to parts of the message. Parsers are global objects most often used together with filters and rewrite rules.
The AxoSyslog application provides the following possibilities to parse the messages, or parts of the messages, as shown on the following list. There are several built-in parsers for application-specific logs.
Note that by default, AxoSyslog parses every message as a syslog message. To disable parsing the message as a syslog message, use the flags(no-parse) option of the source. To explicitly parse a message as a syslog message, use the syslog parser. For details, see Parsing syslog messages.
13.1 - Apache access log parser
The Apache access log parser can parse the access log messages of the Apache HTTP Server. The AxoSyslog application can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The apache-accesslog-parser() supports both the Common Log Format and the Combined Log Format of Apache (for details, see the Apache HTTP Server documentation). The following is a sample log message:
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700]"GET /apache_pb.gif HTTP/1.0"2002326
Starting with version 3.21, virtualhost and the port of the virtualhost (vhost) is also supported, for example:
The parser extracts the following fields from the messages: vhost, port, clientip, ident, auth, timestamp, rawrequest, response, bytes, referrer, and agent. The rawrequest field is further segmented into the verb, request, and httpversion fields. The AxoSyslog apache-accesslog-parser() parser uses the same naming convention as Logstash.
Example: Using the apache-accesslog-parser parser
In the following example, the source is a log file created by an Apache web server. The parser automatically inserts the .apache. prefix before all extracted name-value pairs. The destination is a file that uses the format-json template function. Every name-value pair that begins with a dot (.) character will be written to the file (dot-nv-pairs). The log statement connects the source, the destination, and the parser.
To use this parser, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The apache-accesslog-parser() is actually a reusable configuration snippet configured parse Apache access log messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
13.1.1 - Options of apache-accesslog-parser() parsers
The apache-accesslog-parser() has the following options.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, apache-accesslog-parser() uses the .apache. prefix. To modify it, use the following format:
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
13.2 - Check Point Log Exporter parser
The Check Point Log Exporter parser can parse Check Point log messages. These messages do not completely comply with the syslog RFCs, making them difficult to parse. The checkpoint-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following formats:
If you find a message that the checkpoint-parser() cannot properly parse, contact us, so we can improve the parser.
By default, the Check Point-specific fields are extracted into name-value pairs prefixed with .checkpoint. For example, the action in the previous message becomes ${.checkpoint.action}. You can change the prefix using the prefix option of the parser.
Note that the parser expects that the entire incorrectly formatted syslog message (starting with its <PRI> value) is in $MSG, which you can achieve by using flags(no-parse) on the input driver.
The checkpoint-parser() is actually a reusable configuration snippet configured to parse Check Point messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, checkpoint-parser() uses the .checkpoint. prefix. To modify it, use the following format:
The Cisco parser can parse the log messages of various Cisco devices. The messages of these devices often do not completely comply with the syslog RFCs, making them difficult to parse. The cisco-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs, extracting also the Cisco-specific values, for example, the mnemonic. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse variations of the following message format:
<189>29: foo: *Apr 29 13:58:40.411: %SYS-5-CONFIG_I: Configured from console by console
<190>30: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 stopped - CLI initiated
<190>31: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 started - CLI initiated
<189>32: 0.0.0.0: *Apr 29 13:59:12.491: %SYS-5-CONFIG_I: Configured from console by console
<189>32: foo: *Apr 29 13:58:46.411: %SYSMGR-STANDBY-3-SHUTDOWN_START: The System Manager has started the shutdown procedure.
Note
Not every Cisco log message conforms to this format. If you find a message that the cisco-parser() cannot properly parse, contact us, so we can improve the parser.
The AxoSyslog application normalizes the parsed log messages into the following format:
By default, the Cisco-specific fields are extracted into the following name-value pairs:${.cisco.facility}, ${.cisco.severity}, ${.cisco.mnemonic}. You can change the prefix using the prefix option.
Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The cisco-parser() is actually a reusable configuration snippet configured to parse Cisco messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, cisco-parser() uses the .cisco. prefix. To modify it, use the following format:
parser { cisco-parser(prefix("myprefix."));};
13.4 - Parsing messages with comma-separated and similar values
The AxoSyslog application can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) at delimiter characters or strings to named fields (columns) using the csv (comma-separated-values) parser The parsed fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.
Parsers are similar to filters: they must be defined in the AxoSyslog configuration file and used in the log statement. You can also define the parser inline in the log path.
Note
The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.
To create a csv-parser(), you have to define the columns of the message, the separator characters or strings (also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used to escape the delimiter characters (quote-pairs()).
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
Starting with AxoSyslog version 4.5, you can omit the columns() option, and extract the values into matches ($1, $2, $3, and so on), which are available as the anonymous list $*. For example:
To parse such logs, the delimiter character is set to a single whitespace (delimiters(" ")). Whitespaces between quotes and brackets are ignored (quote-pairs('""[]')).
The results can be used for example, to separate log messages into different files based on the APACHE.USER_NAME field. If the field is empty, the nouser name is assigned.
Multiple parsers can be used to split a part of an already parsed message into further segments. The following example splits the timestamp of a parsed Apache log message into separate fields.
The AxoSyslog application can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) at delimiter characters or strings to named fields (columns) using the csv (comma-separated-values) parser The parsed fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.
Parsers are similar to filters: they must be defined in the AxoSyslog configuration file and used in the log statement. You can also define the parser inline in the log path.
Note
The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.
To create a csv-parser(), you have to define the columns of the message, the separator characters or strings (also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used to escape the delimiter characters (quote-pairs()).
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
columns()
Synopsis:
columns(“PARSER.COLUMN1”, “PARSER.COLUMN2”, …)
Description: Specifies the name of the columns to separate messages to. These names will be automatically available as macros. The values of these macros do not include the delimiters.
Starting with AxoSyslog version 4.5, you can omit the columns() option, and extract the values into matches ($1, $2, $3, and so on), which are available as the anonymous list $*. For example:
Description: The delimiter is the character or string that separates the columns in the message. If you specify multiple characters using the delimiters(chars("<delimiter_characters>)) option, every character will be treated as a delimiter. To separate the columns at the tabulator (tab character), specify \\t. For example, to separate the text at every hyphen (-) and colon (:) character, use delimiters(chars("-:")), Note that the delimiters will not be included in the column values.
String delimiters
If you have to use a string as a delimiter, list your string delimiters in the delimiters(strings("<delimiter_string1>, "<delimiter_string2>> ...)") format.
By default, AxoSyslog uses space as a delimiter. If you want to use only the strings as delimiters, you have to disable the space delimiter, for example: delimiters(chars(""), strings("<delimiter_string>))
Otherwise, AxoSyslog will use the string delimiters in addition to the default character delimiter, so delimiters(strings("==")) actually equals delimiters(chars(" "), strings("==")), and not delimiters(chars(""), strings("=="))
Multiple delimiters
If you use more than one delimiter, note the following points:
AxoSyslog will split the message at the nearest possible delimiter. The order of the delimiters in the configuration file does not matter.
You can use both string delimiters and character delimiters in a parser.
The string delimiters can include characters that are also used as character delimiters.
If a string delimiter and a character delimiter both match at the same position of the message, AxoSyslog uses the string delimiter.
dialect()
Synopsis:
escape-none, escape-backslash, escape-double-char, or escape-backslash-with-sequences
Description: Specifies how to handle escaping in the parsed message. Default value: escape-none
escape-backslash: The parsed message uses the backslash (\\) character to escape quote characters.
escape-backslash-with-sequences: The parsed message uses "" as an escape character but also supports C-style escape
sequences, like \n or \r. Available in AxoSyslog version 4.0 and later.
escape-double-char: The parsed message repeats the quote character when the quote character is used literally. For example, to escape a comma (,), the message contains two commas (,,).
escape-none: The parsed message does not use any escaping for using the quote character literally.
Description: Specifies various options for parsing the message. The following flags are available:
drop-invalid: When the drop-invalid option is set, the parser does not process messages that do not match the parser. For example, a message does not match the parser if it has less columns than specified in the parser, or it has more columns but the greedy flag is not enabled. Using the drop-invalid option practically turns the parser into a special filter, that matches messages that have the predefined number of columns (using the specified delimiters).
Note
Messages dropped as invalid can be processed by a fallback log path. For details on the fallback option, see Log path flags.
greedy: The greedy option assigns the remainder of the message to the last column, regardless of the delimiter characters set. You can use this option to process messages where the number of columns varies.
strip-whitespace: The strip-whitespace flag removes leading and trailing whitespaces from all columns.
Example: Adding the end of the message to the last column
If the greedy option is enabled, AxoSyslog adds the not-yet-parsed part of the message to the last column, ignoring any delimiter characters that may appear in this part of the message.
For example, you receive the following comma-separated message: example 1, example2, example3, and you segment it with the following parser:
The COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, respectively. If the message looks like example 1, example2, example3, some more information, then any text appearing after the third comma (that is, some more information) is not parsed, and possibly lost if you use only the variables to reconstruct the message (for example, to send it to different columns of an SQL table).
Using the greedy flag will assign the remainder of the message to the last column, so that the COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, some more information.
Description: If the value of a column is the value of the null() parameter, AxoSyslog changes the value of the column to an empty string. For example, if the columns of the message contain the “N/A” string to represent empty values, you can use the null("N/A") option to change these values to empty stings.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
This parser does not have a default prefix. To configure a custom prefix, use the following format:
parser { csv-parser(prefix("myprefix."));};
on-type-error()
Synopsis:
string
Description: Specifies what to do when casting a parsed value to a specific data type fails. Note that the flags(drop-invalid) option and the on-error() global option also affects the behavior.
Description: List quote-pairs between single quotes. Delimiter characters or strings enclosed between quote characters are ignored. Note that the beginning and ending quote character does not have to be identical, for example, [} can also be a quote-pair. For an example of using quote-pairs() to parse Apache log files, see Example: Parsing Apache log files.
template()
Synopsis:
template("${<macroname>}")
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
The date parser can extract dates from non-syslog messages. It operates by default on the ${MESSAGE} part of the log message, but can operate on any template or field provided. The parsed date will be available as the sender date (that is, the ${S_DATE}, ${S_ISODATE}, ${S_MONTH}, and so on, and related macros). (To store the parsed date as the received date, use the time-stamp(recvd) option.)
Note
Note that parsing will fail if the format string does not match the entire template or field. Since by default AxoSyslog uses the ${MESSAGE} part of the log message, parsing will fail, unless the log message contains only a date, but that is unlikely, so practically you will have to segment the message (for example, using a csv-parser()) before using the date-parser(). You can also use date-parser() to parse dates received in a JSON or key-value-formatted log message.
In the following example, AxoSyslog parses dates like 01/Jan/2016:13:05:05 PST from a field called MY_DATE using the following format string: format("%d/%b/%Y:%H:%M:%S %Z") (how you create this field from the incoming message is not shown in the example). In the destination template every message will begin with the timestamp in ISODATE format. Since the syslog parser is disabled, AxoSyslog will include the entire original message (including the original timestamp) in the ${MESSAGE} macro.
In the template option, you can use template functions to specify which part of the message to parse with the format string. The following example selects the first 24 characters of the ${MESSAGE} macro.
In AxoSyslog version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:
The date-parser() parser has the following options.
format()
Synopsis:
format(string)
Default:
Description: Specifies the format how AxoSyslog should parse the date. You can use the following format elements:
%% PERCENT
%a day of the week, abbreviated
%A day of the week
%b month abbr
%B month
%c MM/DD/YY HH:MM:SS
%C ctime format: Sat Nov 19 21:05:57 1994 %d numeric day of the month, with leading zeros (eg 01..31) %e like %d, but a leading zero is replaced by a space (eg 1..31) %f microseconds, leading 0's, extra digits are silently discarded
%D MM/DD/YY
%G GPS week number (weeks since January 6, 1980)
%h month, abbreviated
%H hour, 24 hour clock, leading 0's) %I hour, 12 hour clock, leading 0's)
%j day of the year
%k hour
%l hour, 12 hour clock
%L month number, starting with 1
%m month number, starting with 01
%M minute, leading 0's
%n NEWLINE
%o ornate day of month -- "1st", "2nd", "25th", etc.
%p AM or PM
%P am or pm (Yes %p and %P are backwards :) %q Quarter number, starting with 1 %r time format: 09:05:57 PM
%R time format: 21:05
%s seconds since the Epoch, UCT
%S seconds, leading 0's
%t TAB
%T time format: 21:05:57
%U week number, Sunday as first day of week
%w day of the week, numerically, Sunday==0 %W week number, Monday as first day of week
%x date format: 11/19/94
%X time format: 21:05:57
%y year (2 digits) %Y year (4 digits) %Z timezone in ascii format (for example, PST), or in format -/+0000
%z timezone in ascii format (for example, PST), or in format -/+0000 (Required element)
Warning
When using the %z and %Z format elements, consider that while %z strictly expects a specified timezone, and triggers a warning if the timezone is not specified, %Z does not trigger a warning if the timezone is not specified.
For further information about the %z and %Z format elements, see the ‘DESCRIPTION’ section on the srtptime(3) - NetBSD Manual Pages.
For example, for the date 01/Jan/2016:13:05:05 PST use the following format string: format("%d/%b/%Y:%H:%M:%S %Z")
In AxoSyslog version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
flags()
Type:
guess-timezone
Default:
empty string
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp. For example:
date-parser(flags(guess-timezone));
time-stamp()
Synopsis:
stamp or recvd
Default:
stamp
Description: Determines if the parsed date values are treated as sent or received date. If you use time-stamp(stamp), AxoSyslog adds the parsed date to the S_ macros (corresponding to the sent date). If you use time-stamp(recvd), AxoSyslog adds the parsed date to the R_ macros (corresponding to the received date).
time-zone()
Synopsis:
time-zone(string)
Default:
Description: If this option is set, AxoSyslog assumes that the parsed timestamp refers to the specified timezone. The timezone set in the time-zone() option overrides any timezone information parsed from the timestamp.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
value()
Synopsis:
string
Default:
Available in AxoSyslog 4.1 and later.
Description: Instruct date-parser() to store the resulting timestamp in a name-value pair specified in value(), instead of changing the timestamp value of the LogMessage.
13.6 - db-parser: Process message content with a pattern database (patterndb)
13.6.1 - Classifying log messages
The AxoSyslog application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, AxoSyslog is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example, can label the messages as user login, application crash, file transfer, and so on events.
To find the pattern that matches a particular message, AxoSyslog uses a method called longest prefix match radix tree. This means that AxoSyslog creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.
To classify a message, AxoSyslog selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.
To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimal number (for example, 1, 123, 894054, 0xFFFF, and so on). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Using pattern parsers.
The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand AxoSyslog pattens than regular expressions.
Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by AxoSyslog can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2
A regular expression matching this message from the logcheck project: Accepted (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?
An AxoSyslog database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from @QSTRING:client_addr: @port @NUMBER:port:@ ssh2
For details on using pattern databases to classify log messages, see Using pattern databases.
13.6.1.1 - The structure of the pattern database
The pattern database is organized as follows:
The pattern database consists of rulesets. A ruleset consists of a Program Pattern and a set of rules: the rules of a ruleset are applied to log messages if the name of the application that sent the message matches the Program Pattern of the ruleset. The name of the application (the content of the ${PROGRAM} macro) is compared to the Program Patterns of the available rulesets, and then the rules of the matching rulesets are applied to the message. (If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it.)
The Program Pattern can be a string that specifies the name of the application or the beginning of its name (for example, to match for sendmail, the program pattern can be sendmail, or just send), and the Program Pattern can contain pattern parsers. Note that pattern parsers are completely independent from the AxoSyslog parsers used to segment messages. Additionally, every rule has a unique identifier: if a message matches a rule, the identifier of the rule is stored together with the message.
Rules consist of a message pattern and a class. The Message Pattern is similar to the Program Pattern, but is applied to the message part of the log message (the content of the ${MESSAGE} macro). If a message pattern matches the message, the class of the rule is assigned to the message (for example, Security, Violation, and so on).
Rules can also contain additional information about the matching messages, such as the description of the rule, an URL, name-value pairs, or free-form tags.
Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers.
Note
If the ${PROGRAM} part of a message is empty, rules with an empty Program Pattern are used to classify the message.
If the same Program Pattern is used in multiple rulesets, the rules of these rulesets are merged, and every rule is used to classify the message. Note that message patterns must be unique within the merged rulesets, but the currently only one ruleset is checked for uniqueness.
If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it.
13.6.1.2 - How pattern matching works
The followings describe how patterns work. This information applies to program patterns and message patterns alike, even though message patterns are used to illustrate the procedure.
Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers. Pattern parsers attempt to parse a sequence of characters according to certain rules.
Note
Wildcards and regular expressions cannot be used in patterns. The @ character must be escaped, that is, to match for this character, you have to write @@ in your pattern. This is required because pattern parsers are enclosed between @ characters.
When a new message arrives, AxoSyslog attempts to classify it using the pattern database. The available patterns are organized alphabetically into a tree, and AxoSyslog inspects the message character-by-character, starting from the beginning. This approach ensures that only a small subset of the rules must be evaluated at any given step, resulting in high processing speed. Note that the speed of classifying messages is practically independent from the total number of rules.
For example, if the message begins with the Apple string, only patterns beginning with the character A are considered. In the next step, AxoSyslog selects the patterns that start with Ap, and so on, until there is no more specific pattern left. The AxoSyslog application has a strong preference for rules that match the input string completely.
Note that literal matches take precedence over pattern parser matches: if at a step there is a pattern that matches the next character with a literal, and another pattern that would match it with a parser, the pattern with the literal match is selected. Using the previous example, if at the third step there is the literal pattern Apport and a pattern parser Ap@STRING@, the Apport pattern is matched. If the literal does not match the incoming string (for example, Apple), AxoSyslog attempts to match the pattern with the parser. However, if there are two or more parsers on the same level, only the first one will be applied, even if it does not perfectly match the message.
If there are two parsers at the same level (for example, Ap@STRING@ and Ap@QSTRING@), it is random which pattern is applied (technically, the one that is loaded first). However, if the selected parser cannot parse at least one character of the message, the other parser is used. But having two different parsers at the same level is extremely rare, so the impact of this limitation is much less than it appears.
13.6.1.3 - Artificial ignorance
Artificial ignorance is a method used to detect anomalies. When applied to log analysis, it means that you ignore the regular, common log messages — these are the result of the regular behavior of your system, and therefore are not too concerning. However, new messages that have not appeared in the logs before can signal important events, and should be therefore investigated. “By definition, something we have never seen before is anomalous” (Marcus J. Ranum).
The AxoSyslog application can classify messages using a pattern database: messages that do not match any pattern are classified as unknown. This provides a way to use artificial ignorance to review your log messages. You can periodically review the unknown messages — AxoSyslog can send them to a separate destination, and add patterns for them to the pattern database. By reviewing and manually classifying the unknown messages, you can iteratively classify more and more messages, until only the really anomalous messages show up as unknown.
Obviously, for this to work, a large number of message patterns are required. The radix-tree matching method used for message classification is very effective, can be performed very fast, and scales very well. Basically the time required to perform a pattern matching is independent from the number of patterns in the database. For sample pattern databases, see Downloading sample pattern databases.
13.6.2 - Using pattern databases
To classify messages using a pattern database, include a db-parser() statement in your syslog-ng.conf configuration file using the following syntax:
Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.
Example: Defining pattern databases
The following statement uses the database located at /opt/syslog-ng/var/db/patterndb.xml.
By default, AxoSyslog tries to apply the patterns to the body of the incoming messages, that is, to the value of the $MESSAGE macro. If you want to apply patterns to a specific field, or to an expression created from the log message (for example, using template functions or other parsers), use the message-template() option. For example:
By default, AxoSyslog uses the name of the application (content of the ${PROGRAM} macro) to select which rules to apply to the message. If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it. For example:
Note that the program-template() option is available in AxoSyslog version 3.21 and later.
Note
The default location of the pattern database file is /opt/syslog-ng/var/run/patterndb.xml. The file option of the db-parser() statement can be used to specify a different file, thus different db-parser statements can use different pattern databases.
Example: Using classification results
The following destination separates the log messages into different files based on the class assigned to the pattern that matches the message (for example, Violation and Security type messages are stored in a separate file), and also adds the ID of the matching rule to the message:
Note that if you chain pattern databases, that is, use multiple databases in the same log path, the class assigned to the message (the value of ${.classifier.class}) will be the one assigned by the last pattern database. As a result, a message might be classified as unknown even if a previous parser successfully classified it. For example, consider the following configuration:
Even if db_parser1 matches the message, db_parser2 might set ${.classifier.class} to unknown. To avoid this problem, you can use an ‘if’ statement to apply the second parser only if the first parser could not classify the message:
If you want to automatically drop unmatched messages (that is, discard every message that does not match a pattern in the pattern database), use the drop-unmatched() option in the definition of the pattern database:
Note that the drop-unmatched() option is available in AxoSyslog version 3.11 and later.
13.6.2.1 - Using parser results in filters and templates
The results of message classification and parsing can be used in custom filters and templates, for example, in file and database templates. The following built-in macros allow you to use the results of the classification:
The .classifier.class macro contains the class assigned to the message (for example, violation, security, or unknown).
The .classifier.rule_id macro contains the identifier of the message pattern that matched the message.
The .classifier.context_id macro contains the identifier of the context for messages that were correlated. For details on correlating messages, see Correlating log messages using pattern databases.
Example: Using classification results for filtering messages
To filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in a log statement.
Filtering on the unknown class selects messages that did not match any rule of the pattern database. Routing these messages into a separate file allows you to periodically review new or unknown messages.
To filter on messages matching a specific classification rule, create a filter that checks the .classifier.rule_id macro. The unique identifier of the rule (for example, e1e9c0d8-13bb-11de-8293-000c2922ed0a) is the id attribute of the rule in the XML database.
Pattern database rules can assign tags to messages. These tags can be used to select tagged messages using the tags() filter function.
Note
The AxoSyslog application automatically adds the class of the message as a tag using the .classifier.<message-class> format. For example, messages classified as “system” receive the .classifier.system tag. Use the tags() filter function to select messages of a specific class.
The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.
Example: Using pattern parsers as macros
For example, you want to parse messages of an application that look like "Transaction: <type>", where <type> is a string that has different values (for example, refused, accepted, incomplete, and so on). To parse these messages, you can use the following pattern:
'Transaction: @ESTRING::.@'
Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, for example:
'Transaction: @ESTRING:TRANSACTIONTYPE:.@'
After that, add a custom template to the log path that uses this template. For example, to select every accepted transaction, use the following custom filter in the log path:
match("accepted" value("TRANSACTIONTYPE"));
Note
The above macros can be used in database columns and filename templates as well, if you create custom templates for the destination or logspace.
Use a consistent naming scheme for your macros, for example, APPLICATIONNAME_MACRONAME.
13.6.2.2 - Downloading sample pattern databases
To simplify the building of pattern databases, Axoflow has released sample databases. You can download sample pattern databases from the Axoflow GitHub page.
Note that these pattern databases are only samples and experimental databases. They are not officially supported, and may or may not work in your environment.
The pattern databases are available under the Creative Commons Attribution-Share Alike 3.0 (CC by-SA) license. This includes every pattern database written by community contributors or the Axoflow staff. It means that:
You are free to use and modify the patterns for your needs.
If you redistribute the pattern databases, you must distribute your modifications under the same license.
If you redistribute the pattern databases, you must make it obvious that the source of the original pattern databases is the GitHub page.
For legal details, the full text of the license is available here.
If you create patterns that are not available in the GitHub repository, consider sharing them with the community. To do this, open a GitHub issue, or contact us.
13.6.2.3 - Correlating log messages using pattern databases
Log messages are supposed to describe events, but applications often separate information about a single event into different log messages. For example, the Postfix email server logs the sender and recipient addresses into separate log messages, or in case of an unsuccessful login attempt, the OpenSSH server sends a log message about the authentication failure, and the reason of the failure in the next message. Of course, messages that are not so directly related can be correlated as well, for example, login-logout messages, and so on.
To correlate log messages with AxoSyslog, you can add messages into message-groups called contexts. A context consists of a series of log messages that are related to each other in some way, for example, the log messages of an SSH session can belong to the same context. As new messages come in, they may be added to a context. Also, when an incoming message is identified it can trigger actions to be performed, for example, generate a new message that contains all the important information that was stored previously in the context.
There are two attributes for pattern database rules that determine if a message matching the rule is added to a context: context-scope and context-id. The context-scope attribute acts as an early filter, selecting messages sent by the same process (${HOST}${PROGRAM}${PID} is identical), application (${HOST}${PROGRAM} is identical), or host, while the context-id actually adds the message to the context specified in the id. The context-id can be a simple string, or can contain macros or values extracted from the log messages for further filtering. Starting with AxoSyslog version 3.5, if a message is added to a context, AxoSyslog automatically adds the identifier of the context to the .classifier.context_id macro of the message.
Note
Message contexts are persistent and are not lost when AxoSyslog is reloaded (SIGHUP), but are lost when AxoSyslog is restarted.
Another parameter of a rule is the context-timeout attribute, which determines how long a context is stored, that is, how long AxoSyslog waits for related messages to arrive.
Note the following points about timeout values:
When a new message is added to a context, AxoSyslog will restart the timeout using the context-timeout set for the new message.
When calculating if the timeout has already expired or not, AxoSyslog uses the timestamps of the incoming messages, not system time elapsed between receiving the two messages (unless the messages do not include a timestamp, or the keep-timestamp(no) option is set). That way AxoSyslog can be used to process and correlate already existing log messages offline. However, the timestamps of the messages must be in chronological order (that is, a new message cannot be older than the one already processed), and if a message is newer than the current system time (that is, it seems to be coming from the future), AxoSyslog will replace its timestamp with the current system time.
If the context-timeout is 10 seconds and AxoSyslog receives the messages within 1 second, the timeout event will occour immediately, because the difference of the two timestamp (60 seconds) is larger than the timeout value (10 seconds).
Avoid using unnecessarily long timeout values on high-traffic systems, as storing the contexts for many messages can require considerable memory. For example, if two related messages usually arrive within seconds, it is not needed to set the timeout to several hours.
13.6.2.3.1 - Referencing earlier messages of the context
When using the <value> element in pattern database rules together with message correlation, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For example, if there are three log messages in a context, and you are creating a generated message for the third log message, the ${HOST}@1 expression refers to the host field of the current (third) message in the context, the ${HOST}@2 expression refers to the host field of the previous (second) message in the context, ${PID}@3 to the PID of the first message, and so on. For example, the following message can be created from SSH login/logout messages (for details on generating new messages, see Triggering actions for identified messages): An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}.
Warning
When referencing an earlier message of the context, always enclose the field name between braces, for example, ${PID}@3. The reference will not work if you omit the braces.
Note
To use a literal @ character in a template, use @@.
Example: Referencing values from an earlier message
The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):
<actions>
<action>
<message>
<values>
<value name="MESSAGE">An SSH session for${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE} </value>
</values>
</message>
</action>
</actions>
If you do not know in which message of the context contains the information you need, you can use the grep, the context-lookup, or the context-values template functions.
Example: Using the grep template function
The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.
$(grep ("${username}"=="root")${auth_method})
To perform calculations on fields that have numerical values, see Numerical operations.
13.6.3 - Triggering actions for identified messages
The AxoSyslog application can generate (trigger) messages automatically if certain events occur, for example, a specific log message is received, or the correlation timeout of a message expires. Basically, you can define messages for every pattern database rule that are emitted when a message matching the rule is received. Triggering messages is often used together with message correlation, but can also be used separately. When used together with message correlation, you can also create a new correlation context when a new message is received.
The generated message is injected into the same place where the db-parser() statement is referenced in the log path. To post the generated message into the internal() source instead, use the inject-mode() option in the definition of the parser.
Example: Sending triggered messages to the internal() source
To send the generated messages to the internal source, use the inject-mode(internal) option:
The generated message must be configured in the pattern database rule. It is possible to create an entire message, use macros and values extracted from the original message with pattern database, and so on.
Example: Generating messages for pattern database matches
When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.
<actions><action><message><values><valuename="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value></values></message></action></actions>
To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.
Example: Generating messages with inherited values
The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name
In AxoSyslog version 3.8 and newer, you can create a new context as an action. For details, see Element: create-context.
The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.
For details on configuring actions, see the description of the pattern database format.
13.6.3.1 - Conditional actions
To limit when a message is triggered, use the condition attribute and specify a filter expression: the action will be executed only if the condition is met. For example, the following action is executed only if the message was sent by the host called myhost.
The following example triggers different actions based on the number of messages in the context. This way you can check if the context contains enough messages for the event to be complete, and execute a different action if it does not.
To perform an external action when a message is triggered, for example, to send the message in an email, you have to route the generated messages to an external application using the program() destination.
Example: Sending triggered messages to external applications
The following sample configuration selects the triggered messages and sends them to an external script.
Set a field in the triggered message that is easy to identify and filter. For example:
<values><valuename="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value><valuename="TRIGGER">yes</value></values>
Create a destination that will process the triggered messages.
The AxoSyslog application automatically fills the fields for the generated message based on the scope of the context, for example, the HOST and PROGRAM fields if the context-scope is program.
When used together with message correlation, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For details, see Referencing earlier messages of the context.
Example: Referencing values from an earlier message
The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):
<actions>
<action>
<message>
<values>
<value name="MESSAGE">An SSH session for${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE} </value>
</values>
</message>
</action>
</actions>
You can use the name-value pairs of other messages of the context. If you set the inherit-properties attribute of the generated message to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. This means that you can refer to a name-value pair without having to know which message of the context included it. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. To refer to an earlier value, use the @<distance-of-referenced-message-from-the-current> suffix format.
<action><messageinherit-properties='context'>
Example: Using the inherit-properties option
For example, if inherit-properties is set to context, and you have a rule that collects SSH login and logout messages to the same context, you can use the following value to generate a message collecting the most important information form both messages, including the beginning and end date.
<valuename="MESSAGE">An SSH session for ${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid: $PID.</value>
The following is a detailed rule for this purpose.
<patterndbversion='4'pub_date='2015-04-13'><rulesetname='sshd'id='12345678'><pattern>sshd</pattern><rules><!-- The pattern database rule for the first log message --><ruleprovider='me'id='12347598'class='system'context-id="ssh-login-logout"context-timeout="86400"context-scope="process"><!-- Note the context-id that groups together the
relevant messages, and the context-timeout value that
determines how long a new message can be added to the
context --><patterns><pattern>Accepted @ESTRING:SSH.AUTH_METHOD: @for @ESTRING:SSH_USERNAME: @from @ESTRING:SSH_CLIENT_ADDRESS: @port @ESTRING:: @@ANYSTRING:SSH_SERVICE@</pattern><!-- This is the actual pattern used to identify
the log message. The segments between the @
characters are parsers that recognize the variable
parts of the message - they can also be used as
macros. --></patterns></rule><!-- The pattern database rule for the fourth log message --><ruleprovider='me'id='12347599'class='system'context-id="ssh-login-logout"context-scope="process"><patterns><pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:SSH_USERNAME@</pattern></patterns><actions><action><messageinherit-properties='context'><values><valuename="MESSAGE">An SSH session for ${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid: $PID.</value><valuename="TRIGGER">yes</value><!-- This is the new log message
that is generated when the logout
message is received. The macros ending
with @2 reference values of the
previous message from the context. --></values></message></action></actions></rule></rules></ruleset></patterndb>
It is possible to generate a message when the context-timeout of the original message expires and no new message is added to the context during this time. To accomplish this, include the trigger="timeout" attribute in the action element:
<action trigger="timeout">
Example: Sending alert when a client disappears
The following example shows how to combine various features of AxoSyslog to send an email alert if a client stops sending messages.
Configure your clients to send MARK messages periodically. It is enough to configure MARK messages for the destination that forwards your log messages to your AxoSyslog server (mark-mode(periodical)).
On your AxoSyslog server, create a pattern database rule that matches on the incoming MARK messages. In the rule, set the context-scope attribute to host, and the context-timeout attribute to a value that is higher than the mark-freq value set on your clients (by default, mark-freq is 1200 seconds, so set context-timeout at least to 1500 seconds, but you might want to use a higher value, depending on your environment).
Add an action to this rule that sends you an email alert if the context-timeout expires, and the server does not receive a new MARK message (<action trigger="timeout">).
On your AxoSyslog server, use the pattern database in the log path that handles incoming log messages.
13.6.4 - Creating pattern databases
13.6.4.1 - Using pattern parsers
Pattern parsers attempt to parse a part of the message using rules specific to the type of the parser. Parsers are enclosed between @ characters. The syntax of parsers is the following:
a beginning @ character,
the type of the parser written in capitals,
optionally a name,
parameters of the parser, if any, and
a closing @ character.
Example: Pattern parser syntax
A simple parser:
@STRING@
A named parser:
@STRING:myparser_name@
A named parser with a parameter:
@STRING:myparser_name:*@
A parser with a parameter, but without a name:
@STRING::*@
Patterns and literals can be mixed together. For example, to parse a message that begins with the Host: string followed by an IP address (for example, Host: 192.168.1.1), the following pattern can be used: Host:@IPv4@.
Note
Note that using parsers is a CPU-intensive operation. Use the ESTRING and QSTRING parsers whenever possible, as these can be processed much faster than the other parsers.
Example: Using the STRING and ESTRING parsers
For example, look at the following message: user=joe96 group=somegroup.
@STRING:mytext:@ parses only to the first non-alphanumeric character (=), parsing only user, so the value of the ${mytext} macro will be user
@STRING:mytext:=@ parses the equation mark as well, and proceeds to the next non-alphanumeric character (the whitespace), resulting in user=joe96
@STRING:mytext:= @ will parse the whitespace as well, and proceed to the next non-alphanumeric non-equation mark non-whitespace character, resulting in user=joe96 group=somegroup
Of course, usually it is better to parse the different values separately, like this: "user=@STRING:user@ group=@STRING:group@".
If the username or the group may contain non-alphanumeric characters, you can either include these in the second parameter of the parser (as shown at the beginning of this example), or use an ESTRING parser to parse the message till the next whitespace: "user=@ESTRING:user: @group=@ESTRING:group: @".
13.6.4.1.1 - Pattern parsers
The following parsers are available in AxoSyslog. The internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair. For details on data types, see Specifying data types in value-pairs.
@ANYSTRING@
Parses everything to the end of the message, you can use it to collect everything that is not parsed specifically to a single macro. In that sense its behavior is similar to the greedy() option of the CSV parser.
@DOUBLE@
An obsolete alias of the @FLOAT@ parser.
@EMAIL@
This parser matches an email address. The parameter is a set of characters to strip from the beginning and the end of the email address. That way email addresses enclosed between other characters can be matched easily (for example, <[email protected]> or "[email protected]". Characters that are valid for a hostname are not stripped from the end of the hostname. This includes a trailing period if present.
This parser has a required parameter that acts as the stopcharacter: the parser parses everything until it finds the stopcharacter. For example, to stop by the next " (double quote) character, use @ESTRING::"@. You can use the colon (:) as stopcharacter as well, for example: @ESTRING:::@. You can also specify a stopstring instead of a single character, for example, @ESTRING::stop_here.@. The @ character cannot be a stopcharacter, nor can line-breaks or tabs.
@FLOAT@
A floating-point number that may contain a dot (.) character. (Up to AxoSyslog 3.1, the name of this parser was @DOUBLE@.)
@HOSTNAME@
Parses a generic hostname. The hostname may contain only alphanumeric characters (A-Z,a-z,0-9), hypen (-), or dot (.).
@IPv4@
Parses an IPv4 IP address (numbers separated with a maximum of 3 dots).
@IPv6@
Parses any valid IPv6 IP address.
@IPvANY@
Parses any IP address.
@LLADDR@
Parses a Link Layer Address in the xx:xx:xx:... form, where each xx is a 2 digit HEX number (an octet). The parameter specifies the maximum number of octets to match and defaults to 20. The MACADDR parser is a special wrapper using the LLADDR parser. For example, the following parser parses maximally 10 octets, and stores the results in the link-level-address macro:
@LLADDR:link-level-address:10@
@MACADDR@
Parses the standard format of a MAC-48 address, consisting of is six groups of two hexadecimal digits, separated by colons. For example, 00:50:fc:e3:cd:37.
@NLSTRING@
This parser parses everything until the next new-line character (more precisely, until the next Unix-style LF or Windows-style CRLF character). For single-line messages, NLSTRING is equivalent with ANYSTRING. For multi-line messages, NLSTRING parses to the end of the current line, while ANYSTRING parses to the end of the message. Using NLSTRING is useful when parsing multi-line messages, for example, Windows logs. For example, the following pattern parses information from Windows security auditing logs.
<pattern>Example-PC\Example: Security Microsoft Windows security auditing.: [Success Audit] A new process has been created.
Subject:
Security ID: @LNSTRING:.winaudit.SubjectUserSid@
Account Name: @LNSTRING:.winaudit.SubjectUserName@
Account Domain: @LNSTRING:.winaudit.SubjectDomainName@
Logon ID: @LNSTRING:.winaudit.SubjectLogonId@
Process Information:
New Process ID: @LNSTRING:.winaudit.NewProcessId@
New Process Name: @LNSTRING:.winaudit.NewProcessName@
Token Elevation Type: @LNSTRING:.winaudit.TokenElevationType@
Creator Process ID: @LNSTRING:.winaudit.ProcessId@
Process Command Line: @LNSTRING:.winaudit.CommandLine@
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.</pattern>
@NUMBER@
A sequence of decimal (0-9) numbers (for example, 1, 0687, and so on). Note that if the number starts with the 0x characters, it is parsed as a hexadecimal number, but only if at least one valid character follows 0x. A leading hyphen (-) is accepted for non-hexadecimal numbers, but other separator characters (for example, dot or comma) are not. To parse floating-point numbers, use the @FLOAT@ parser.
@OPTIONALSET@
Parse any combination of the specified characters. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings (for example, log messages of the Squid application have whitespace padding after the username).
For example, the @OPTIONALSET:: "@ parser will parse any combination of whitespaces and double-quotes.
Available in 3.31 and later.
Note
The @OPTIONALSET@ parser works almost exactly like the @SET@ parser, but the @OPTIONALSET@ parser continues parsing even if none of the specified characters are found.
@PCRE@
Use Perl-Compatible Regular Expressions (as implemented by the PCRE library), after the identification of the potential patterns has happened by the radix implementation.
Syntax: @PCRE:name:regexp@
@QSTRING@
Parse a string between the quote characters specified as parameter. Note that the quote character can be different at the beginning and the end of the quote, for example: @QSTRING::"@ parses everything between two quotation marks ("), while @QSTRING:\<\>@ parses from an opening bracket to the closing bracket. The @ character cannot be a quote character, nor can line-breaks or tabs.
@SET@
Parse any combination of the specified characters until another character is found. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings (for example, log messages of the Squid application have whitespace padding after the username).
For example, the @SET:: "@ parser will parse any combination of whitespaces and double-quotes.
Available in AxoSyslog 3.4 and later.
@STRING@
A sequence of alphanumeric characters (0-9, A-z), not including any whitespace. Optionally, other accepted characters can be listed as parameters (for example, to parse a complete sentence, add the whitespace as parameter, like: @STRING:: @). Note that the @ character cannot be a parameter, nor can line-breaks or tabs.
13.6.4.2 - What's new in the pattern database format V5
The V5 database format has the following differences compared to the V4 format:
The <ruleset> element can now store multiple reference URLs using the new <rule_urls> element. For details, see Element: ruleset.
In an <action>, you can now initialize a new context. As a result, the <message> element is not required. For details, see Element: create-context.
The inherit-properties attribute is deprecated, use the inherit-mode attribute instead. For details, see Element: action.
13.6.4.3 - The pattern database format
Pattern databases are XML files that contain rules describing the message patterns. For sample pattern databases, see Downloading sample pattern databases.
The following scheme describes the V5 format of the pattern database. This format is backwards-compatible with the earlier formats.
Use the pdbtool utility that is bundled with AxoSyslog to test message patterns and convert existing databases to the latest format. For details, see The pdbtool manual page.
To automatically create an initial pattern database from an existing log file, use the pdbtool patternize command. For details, see The pdbtool manual page.
Example: A pattern database containing a single rule
The following pattern database contains a single rule that matches a log message of the ssh application. A sample log message looks like:
Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2
The following is a simple pattern database containing a matching rule.
Note that the rule uses macros that refer to parts of the message, for example, you can use the ${SSH_USERNAME} macro refer to the username used in the connection.
The following is the same example, but with a test message and test values for the parsers.
A container element to group log patterns for an application or program. A <patterndb> element may contain any number of <ruleset> elements.
Attributes
name: The name of the application. Note that the function of this attribute is to make the database more readable, syslog-ng uses the <pattern> element to identify the applications sending log messages.
id: A unique ID of the application, for example, the md5 sum of the name attribute.
Children
patterns
rules
actions
tags
description: OPTIONAL — A description of the ruleset or the application.
url: OPTIONAL — An URL referring to further information about the ruleset or the application.
rule_urls: OPTIONAL — To list multiple URLs referring to further information about the ruleset or the application, enclose the <url> elements into an <urls> element.
A container element. A <patterns> element may contain any number of `elements.
Attributes
N/A
Children
pattern: The name of the application — syslog-ng matches this value to the ${PROGRAM} header of the syslog message to find the rulesets applicable to the syslog message.
Specifying multiple patterns is useful if two or more applications have different names (that is, different ${PROGRAM} fields), but otherwise send identical log messages.
It is not necessary to use multiple patterns if only the end of the ${PROGRAM} fields is different, use only the beginning of the ${PROGRAM} field as the pattern. For example, the Postfix email server sends messages using different process names, but all of them begin with the postfix string.
You can also use parsers in the program pattern if needed, and use the parsed results later. For example: `stfix\@ESTRING:.postfix.component:[@»
Note
If the `element of a ruleset is not specified, AxoSyslog will use this ruleset as a fallback ruleset: it will apply the ruleset to messages that have an empty PROGRAM header, or if none of the program patterns matched the PROGRAM header of the incoming message.
An element containing message patterns and how a message that matches these patterns is classified.
Note
If the following characters appear in the message, they must be escaped in the rule as follows:
@: Use @@, for example, user@@example.com
<: Use \<
>: Use \>
&: Use \&
The element may contain any number of elements.
Attributes
provider: The provider of the rule. This is used to distinguish between who supplied the rule, that is, if it has been created by Axoflow, or added to the XML by a local user.
id: The globally unique ID of the rule.
class: The class of the rule — this class is assigned to the messages matching a pattern of this rule.
An element containing the patterns of the rule. If a element contains multiple *lements, the class of the *»>ssigned to every syslog message matching any of the patterns.
Attributes
N/A
Children
pattern: A pattern describing a log message. This element is also called message pattern. For example:
<pattern>+ ??? root-</pattern>
Note
Support for XML entities is limited, you can use only the following entities: \& \< \> \" \'. User-defined entities are not supported.
description: OPTIONAL — A description of the pattern or the log message matching the pattern.
OPTIONAL — Name-value pairs that are assigned to messages matching the patterns, for example, the representation of the event in the message according to the Common Event Format (CEF) or Common Event Exchange (CEE). The names can be used as macros to reference the assigned values.
Attributes
N/A
Children
value: OPTIONAL — Contains the value of the name-value pair that is assigned to the message.
The <value> element of name-value pairs can include template functions. For details, see Using template functions, for examples, see if.
You can associate types with values using the "type" attribute, for example, integer is a type-cast that associates $foobar with an integer type. For details on data types, see Specifying data types in value-pairs.
<valuename="foobar"type="integer">$PID</value>
db-parser()’s internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair.
When used together with message correlation, the <value> element of name-value pairs can include references to the values of earlier messages from the same context. For details, see Correlating log messages using pattern databases.
name: The name of the name-value pair. It can also be used as a macro to reference the assigned value.
OPTIONAL — A container element for actions that are performed if a message is recognized by the pattern. For details on actions, see Triggering actions for identified messages.
Attributes
N/A
Children
action
Example
Example: Generating messages for pattern database matches
When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.
<actions><action><message><values><valuename="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value></values></message></action></actions>
To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.
Example: Generating messages with inherited values
The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name
OPTIONAL — A container element describing an action that is performed when a message matching the rule is received.
Attributes
condition: An AxoSyslog filter expression. The action is performed only if the message matches the filter. The filter can include macros and name-value pairs extracted from the message. When using actions together with message-correlation, you can also use the $(context-length) macro, which returns the number of messages in the current context. For example, this can be used to determine if the expected number of messages has arrived to the context: condition='"$(context-length)" > "5"'
rate: Specifies maximum how many messages should be generated in the specified time period in the following format: <number-of-messages>/<period-in-seconds>. For example: 1/60 allows 1 message per minute. Rates apply within the scope of the context, that is, if context-scope="host" and rate="1/60", then maximum one message is generated per minute for every host that sends a log message matching the rule. Excess messages are dropped. Note that when applying the rate to the generated messages, AxoSyslog uses the timestamps of the log messages, similarly to calculating the context-timeout. That way rate is applied correctly even if the log messages are processed offline.
trigger: Specifies when the action is executed. The trigger attribute has the following possible values:
match: Execute the action immediately when a message matching the rule is received.
timeout: Execute the action when the correlation timer (context-timeout) of the pattern database rule expires. This is available only if actions are used together with correlating messages.
Children
create-context
message: A container element storing the message to be sent when the action is executed. Currently AxoSyslog sends these messages to the internal() destination.
inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.
context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.
last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.
none: An empty message is created, without inheriting any tags or name-value pairs.
This option is available in AxoSyslog 3.8 and later.
inherit-properties: This attribute is deprecated. Use the inherit-mode attribute instead.
If set to TRUE, the original message that triggered the action is cloned, including its name-value pairs and tags.
If set to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.
This option is available in AxoSyslog 5.3.2 and later.
values: A container element for values and fields that are used to create the message generated by the action.
value: Sets the value of the message field specified in the name attribute of the element. For example, to specify the body of the generated message, use the following syntax:
<value name="MESSAGE">A log message matched rule number $.classifier.rule_id</value>
Note that currently it is not possible to add DATE, FACILITY, or SEVERITY fields to the message.
When the action is used together with message correlation, the AxoSyslog application automatically adds fields to the message based on the context-scope parameter. For example, using context-scope="process" automatically fills the HOST, PROGRAM, and PID fields of the generated message.
name: Name of the message field set by the value element.
Example: Generating messages for pattern database matches
When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.
<actions><action><message><values><valuename="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value></values></message></action></actions>
To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.
Example: Generating messages with inherited values
The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name
OPTIONAL — Creates a new correlation context from the current message and its associated context. This can be used to “split” a context.
Available in AxoSyslog version 3.8 and later.
Attributes
Children
message: A container element storing the message that is added to the new context when the action is executed.
inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.
context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.
last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.
none: An empty message is created, without inheriting any tags or name-value pairs.
The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.
OPTIONAL — An element containing custom keywords (tags) about the messages matching the patterns. The tags can be used to label specific events (for example, user logons). It is also possible to filter on these tags later (for details, see Tagging messages). Starting with AxoSyslog 3.2, the list of tags assigned to a message can be referenced with the ${TAGS} macro.
Attributes
N/A
Children
tag: OPTIONAL — A keyword or tags applied to messages matching the rule.
<examples><example><test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message><test_values><test_valuename="SSH_AUTH_METHOD">password</test_value><test_valuename="SSH_USERNAME">sampleuser</test_value><test_valuename="SSH_CLIENT_ADDRESS">10.50.0.247</test_value><test_valuename="SSH_PORT_NUMBER"type="integer">42156</test_value></test_values></example></examples>
OPTIONAL — A container element for sample log messages that should be recognized by the pattern. These messages can be used also to test the patterns and the parsers.
Attributes
N/A
Children
example
Example
<examples><example><test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message><test_values><test_valuename="SSH.AUTH_METHOD">password</test_value><test_valuename="SSH_USERNAME">sampleuser</test_value><test_valuename="SSH_CLIENT_ADDRESS">10.50.0.247</test_value><test_valuename="SSH_PORT_NUMBER">42156</test_value></test_values></example></examples>
13.7 - Parsing enterprise-wide message model (EWMM) messages
The ewmm-parser() can be used to parse messages sent by another AxoSyslog host using the enterprise-wide message model (EWMM) format. Available in version 3.16 and later. Note that usually you do not have to use this parser directly, because the default-network-drivers() source automatically parses such messages.
Declaration:
parser parser_name { ewmm-parser();};
13.8 - Fortigate parser
The Fortigate parser can parse the log messages of FortiGate/FortiOS (Fortigate Next-Generation Firewall (NGFW)). These messages do not completely comply with the syslog RFCs, making them difficult to parse. The fortigate-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs, see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:
If you find a message that the fortigate-parser() cannot properly parse, contact us, so we can improve the parser.
By default, the Fortigate-specific fields are extracted into name-value pairs prefixed with .fortigate. For example, the devname in the previous message becomes ${.fortigate.devname}. You can change the prefix using the prefix option of the parser.
Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The fortigate-parser() is actually a reusable configuration snippet configured to parse Fortigate messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, websense-parser() uses the .websense. prefix. To modify it, use the following format:
parser { websense-parser(prefix("myprefix."));};
13.8.1 - Fortigate parser options
The fortigate-parser() has the following options:
prefix()
Synopsis:
prefix()
Default:
“.panos.”
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, fortigate-parser() uses the .fortigate. prefix. To modify it, use the following format:
parser { fortigate-parser(prefix("myprefix."));};
template()
Synopsis:
template("${<macroname>}")
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
13.9 - group-lines parser
Available in AxoSyslog version 4.2 and newer.
The group-lines() parser correlates multi-line messages received as separate, but subsequent lines into a single log message. AxoSyslog first collects the received messages into streams of related messages (based on the key() parameter), then grouped into correlation contexts up to timeout() seconds long. Multi-line messages are then identified within these contexts.
Description: Specifies a template that determines which messages form a single stream. Messages where the template expansion results in the same key are considered part of the same stream. Using the key() option, you can extract multi-line messages even if different streams are interleaved in your input.
line-separator()
Type:
string
Default:
\n
Description: In case a multi-line message is found, this string is inserted between the of the new multi-line message. Defaults to the newline character.
multi-line-garbage()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.
When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.
To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.
Warning
If the multi-line-garbage() option is set, AxoSyslog discards lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix().
Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:
indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.
prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.
prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.
The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.
smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.
smart mode is available in AxoSyslog version 4.2 and newer.
The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
multi-line-prefix()
Type:
regular expression starting with the ^ character
Default:
empty string
Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.
Note
To format multi-line messages to your individual needs, consider the following:
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
Example: Processing Tomcat logs
The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.
2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.net.BindException: Address already in use null:8080
at org.apache.catalina.connector.Connector.start(Connector.java:1138) at org.apache.catalina.core.StandardService.start(StandardService.java:531) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177) 2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
INFO: Server startup in 1206 ms
2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:
Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.
multi-line-suffix()
Type:
regular expression
Default:
empty string
Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.
To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.
scope()
Type:
process, program, host, or global
Default:
global
Description: Specifies which messages belong to the same context. The following values are available:
process: Only messages that are generated by the same process of a client belong to the same context, that is, messages that have identical ${HOST}, ${PROGRAM} and ${PID} values.
program: Messages that are generated by the same application of a client belong to the same context, that is, messages that have identical ${HOST} and ${PROGRAM} values.
host: Every message generated by a client belongs to the same context, only the ${HOST} value of the messages must be identical.
global: Every message belongs to the same context. This is the default value.
template()
Type:
template
Default:
Description: A template string that specifies what constitutes an line to group-lines(). In simple cases this is ${MSG} or ${RAWMSG}.
13.10 - iptables parser
The iptables parser can parse the log messages of the iptables command. Available in version 3.16 and later.
The iptables-parser() is actually a reusable configuration snippet configured to parse iptables messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, iptables-parser() uses the .iptables. prefix. To modify it, use the following format:
parser { iptables-parser(prefix("myprefix."));};
13.11 - JSON parser
JavaScript Object Notation (JSON) is a text-based open standard designed for human-readable data interchange. It is used primarily to transmit data between a server and web application, serving as an alternative to XML. It is described in RFC 4627. The AxoSyslog application can separate parts of incoming JSON-encoded log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs.
You can refer to the separated parts of the JSON message using the key of the JSON object as a macro. For example, if the JSON contains {"KEY1":"value1","KEY2":"value2"}, you can refer to the values as ${KEY1} and ${KEY2}. If the JSON content is structured, AxoSyslog converts it to dot-notation-format. For example, to access the value of the following structure {"KEY1": {"KEY2": "VALUE"}}, use the ${KEY1.KEY2} macro.
Warning
If the names of keys in the JSON content are the same as the names of AxoSyslog soft macros, the value from the JSON content will overwrite the value of the macro. For example, the {"PROGRAM":"value1","MESSAGE":"value2"} JSON content will overwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use the prefix() option.
Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see Hard versus soft macros.
Note
When using the json-parser(), AxoSyslog converts all elements of the JSON object to name-value pairs. Any type information originally present in the incoming JSON object is retained, and automatically propagated to other AxoSyslog components (for example, a destination) if they support types.
Elements without a type are treated as strings.
JSON lists (arrays) are converted to AxoSyslog lists, so you can manipulate them using the $(list-*) template functions.
Note that prior to version 4.0, AxoSyslog handled every data as strings.
The JSON parser discards messages if it cannot parse them as JSON messages, so it acts as a JSON-filter as well.
To create a JSON parser, define a parser that has the json-parser() option. Defining the prefix and the marker are optional. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message with the JSON parser, use the template() option. You can also define the parser inline in the log path.
In the following example, the source is a JSON encoded log message. The syslog parser is disabled, so that AxoSyslog does not parse the message: flags(no-parse). The json-parser inserts “.json.” prefix before all extracted name-value pairs. The destination is a file that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.
Description: Extract only the specified subtree from the JSON message. Use the dot-notation to specify the subtree. The rest of the message will be ignored. For example, assuming that the incoming object is named msg, the json-parser(extract-prefix("foo.bar[5]")); parser is equivalent to the msg.foo.bar[5] javascript code. Note that the resulting expression must be a JSON object in order to extract its members into name-value pairs.
This feature also works when the top-level object is an array, because you can use an array index at the first indirection level, for example: json-parser(extract-prefix("[5]")), which is equivalent to msg[5].
In addition to alphanumeric characters, the key of the JSON object can contain the following characters: \!"#$%&'()\*+,-/:;<=>?@\\^_{|}~`
It cannot contain the following characters: .[]
Example: Convert logstash eventlog format v0 to v1
The following parser converts messages in the logstash eventlog v0 format to the v1 format.
Description: The delimiter character to use when parsing flattened keys. Supports Only single characters.
marker
Synopsis:
marker()
Description: Use a marker in case of mixed log messages, to identify JSON encoded messages for the parser.
Some logging implementations require a marker to be set before the JSON payload. The JSON parser is able to find these markers and parse the message only if it is present.
Example: Using the marker option in JSON parser
This json parser parses log messages which use the “@cee:” marker in front of the json payload. It inserts “.cee.” in front of the name of name-value pairs, so later on it is easier to find name-value pairs that were parsed using this parser. (For details on selecting name-value pairs, see value-pairs().)
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
This parser does not have a default prefix. To configure a custom prefix, use the following format:
parser { json-parser(prefix("myprefix."));};
template()
Synopsis:
template("${<macroname>}")
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
13.12 - Parsing key=value pairs
The AxoSyslog application can separate a message consisting of whitespace or comma-separated key=value pairs (for example, Postfix log messages) into name-value pairs. You can also specify other separator character instead of the equal sign, for example, colon (:) to parse MySQL log messages. The AxoSyslog application automatically trims any leading or trailing whitespace characters from the keys and values, and also parses values that contain unquoted whitespace. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs.
You can refer to the separated parts of the message using the key of the value as a macro. For example, if the message contains KEY1=value1,KEY2=value2, you can refer to the values as ${KEY1} and ${KEY2}.
Note
If a log message contains the same key multiple times (for example, key1=value1, key2=value2, key1=value3, key3=value4, key1=value5), then AxoSyslog stores only the last (rightmost) value for the key. Using the previous example, AxoSyslog will store the following pairs: key1=value5, key2=value2, key3=value4.
Warning
If the names of keys in the message is the same as the names of AxoSyslog soft macros, the value from the parsed message will overwrite the value of the macro. For example, the PROGRAM=value1, MESSAGE=value2 content will overwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use the prefix() option.
Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see Hard versus soft macros.
The parser discards message sections that are not key=value pairs, even if they appear between key=value pairs that can be parsed.
The names of the keys can contain only the following characters: numbers (0-9), letters (a-z,A-Z), underscore (_), dot (.), hyphen (-). Other special characters are not permitted.
To parse key=value pairs, define a parser that has the kv-parser() option. Defining the prefix is optional. By default, the parser will process the ${MESSAGE} part of the log message. You can also define the parser inline in the log path.
Declaration:
parser parser_name { kv-parser( prefix());};
Example: Using a key=value parser
In the following example, the source is a log message consisting of comma-separated key=value pairs, for example, a Postfix log message:
The kv-parser inserts the “.kv.” prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.
Description: Specifies the name-value pair where AxoSyslog stores any stray words that appear before or between the parsed key-value pairs (mainly when the pair-separator() option is also set). If multiple stray words appear in a message, then AxoSyslog stores them as a comma-separated list. Note that the prefix() option does not affect the name-value pair storing the stray words. Default value:N/A
Example: Extracting stray words in key-value pairs
This is a list of key-value pairs, where the value separator is = and the pair separator is ;. However, before the last key-value pair (policy=370), there are two stray words: interzone-emtn_s1_vpn-enodeb_om inbound. If you want to store or process these, specify a name-value pair to store them in the extract-stray-words-into() option, for example, extract-stray-words-into("my-stray-words"). The value of ${my-stray-words} for this message will be interzone-emtn_s1_vpn-enodeb_om, inbound
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, kv-parser() uses the .kv. prefix. To modify it, use the following format:
parser { kv-parser(prefix("myprefix."));};
pair-separator()
Synopsis:
pair-separator(")
Description: Specifies the character or string that separates the key-value pairs from each other. Default value: , .
For example, to parse key1=value1;key2=value2 pairs, use kv-parser(pair-separator(";")); .
template()
Synopsis:
template("${<macroname>}")
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
value-separator()
Synopsis:
value-separator("")
Description: Specifies the character that separates the keys from the values. Default value: =.
For example, to parse key:value pairs, use kv-parser(value-separator(":"));.
13.13 - Linux audit parser
The Linux audit parser can parse the log messages of the Linux Audit subsystem (auditd). The AxoSyslog application can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The following is a sample log message of auditd:
Certain fields of the audit log can be encoded in hexadecimal format, for example, the arch field, or the a<number> fields in the previous example. The AxoSyslog application automatically decodes these fields (for example, the c000003e value becomes x86_64).
The AxoSyslog application extracts every field into name-value pairs. It automatically decodes the following fields:
name
proctitle
path
dir
comm
ocomm
data
old
new
To parse the log messages of the Linux Audit subsystem, define a parser that has the linux-audit-parser() option. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message, use the template() option. You can also define the parser inline in the log path.
In the following example, the source is a log file created by auditd. Since the audit log format is not a syslog format, the syslog parser is disabled, so that AxoSyslog does not parse the message: flags(no-parse). The parser inserts “.auditd.” prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination, and the parser.
The linux-audit-parser() has the following options.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, linux-audit-parser() uses the .auditd. prefix. To modify it, use the following format:
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
13.14 - MariaDB parser
The MariaDB parser can parse the log messages of the MariaDB Audit Plugin. The parser supports the syslog output typess’ format. Available in version 3.37 and later.
The mariadb-audit is a reusable configuration snippet configured to parse MariaDB Audit Plugin messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, mariadb-audit uses the .mariadb. prefix. To modify it, use the following format:
metrics-probe() is a special parser that counts the messages that pass through the log path, and creates labeled stats counters based on the fields of the passing messages.
You can configure the name of the keys and the labels. Note that the keys are automatically prefixes with the syslogng_ string. You can use templates in the values of the labels.
The minimal configuration creates counters with the key syslogng_classified_events_total and labels app, host, program and source. For example:
You can query the metrics by running the following command:
syslog-ng-ctl stats prometheus
For example, the following metrics-probe() parser creates a counter called syslogng_custom_key that counts messages that have their custom_label_name_1 field set to foobar, and for these messages it creates separate counters based on the value of the custom_label_name_2 field.
Starting with AxoSyslog 4.4, you can create dynamic labels as well.
Options
increment()
Type:
integer or template
Default:
1
Available in AxoSyslog version 4.2 and newer.
Sets a template, which resolves to a number that defines the increment of the counter. The following example defines a counter called syslogng_input_event_bytes_total, and increases its value with the size of the incoming message (in bytes).
The name of the counter to create. Note that the value of this option is always prefixed with syslogng_, so for example key("my-custom-key") becomes syslogng_my-custom-key.
labels()
Type:
Default:
See the description
The labels used to create separate counters, based on the fields of the messages processed by metrics-probe(). Use the following format:
Note: Drivers configured with internal(yes) register their metrics on level 3. That way if you are creating an SCL, you can disable the built-in metrics of the driver, and create metrics manually using metrics-probe().
13.16 - Netskope parser
The Netskope parser can parse Netskope log messages. These messages do not completely comply with the syslog RFCs, making them difficult to parse. The netskope-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:
If you find a message that the netskope-parser() cannot properly parse, contact us, so we can improve the parser.
The AxoSyslog application sets the ${PROGRAM} field to Netskope.
By default, the Netskope-specific fields are extracted into name-value pairs prefixed with .netskope. For example, the organization_unit in the previous message becomes ${.netskope.organization_unit}. You can change the prefix using the prefix option of the parser.
Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The netskope-parser() is actually a reusable configuration snippet configured to parse Netskope messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, netskope-parser() uses the .netskope. prefix. To modify it, use the following format:
parser { netskope-parser(prefix("myprefix."));};
13.17 - Parse OpenTelemetry messages
By default, AxoSyslog doesn’t parse the fields of incoming OpenTelemetry messages into name-value pairs, but are only available for forwarding using the opentelemetry() destination. To parse the fields into name-value pairs, use the opentelemetry() parser.
The opentelemetry() parser parses the fields into name-value pairs starting with the .otel. prefix.
The type of the message is stored in the .otel.type field (possible values: log, metric, and span).
Resource information is mapped into the .otel.resource.<...> , for example, .otel.resource.dropped_attributes_count, or .otel.resource.schema_url.
Scope information is mapped into .otel.scope.<...>, for example, .otel.scope.name, .otel.scope.schema_url.
The fields of log records are mapped into .otel.log.<...>, for example, .otel.log.body, .otel.log.severity_text.
The fields of metrics are mapped into .otel.metric.<...>, for example, .otel.metric.name, .otel.metric.unit.
The type of the metric is mapped into .otel.metric.data.type. Possible values: gauge, sum, histogram, exponential_histogram, summary.
The actual data is mapped into .otel.metric.data.<type>.<...>, for example, .otel.metric.data.gauge.data_points.0.time_unix_nano.
The fields of traces are mapped into .otel.span.<...>, for example, .otel.span.name, .otel.span.trace_state. Repeated fields have an index, for example, .otel.span.events.5.time_unix_nano.
String, bool, int64, double, and bytes values are mapped to their respective AxoSyslog name-value type, for example, .otel.resource.attributes.string_key becomes a string value.
The mapping of AnyValue type fields is limited.
ArrayValue and KeyValueList types are stored serialized with protobuf type. Note that protobuf and bytes types are only available, unless explicitly type cast. For example, bytes(${.otel.log.span_id}). When using template functions, use --include-bytes, for example, $(format-json .otel.* --include-bytes. In the case of $(format-json), the content is base64-encoded into the bytes content.
The PAN-OS (a short version of Palo Alto Networks Operating System) parser can parse log messages originating from Palo Alto Networks devices. Even though these messages completely comply to the RFC standards, their MESSAGE part is not a plain text. Instead, the MESSAGE part contains a data structure that requires additional parsing.
The panos-parser() of AxoSyslog solves this problem, and can separate PAN-OS log messages to name-value pairs.
PAN-OS log messages from Palo Alto Networks devices.
Limitations
The panos-parser() only works on AxoSyslog version 3.29 or later.
Configuration
You can include the panos-parser() in your AxoSyslog configuration like this:
parser p_parser{ panos-parser();};
To use this parser, the scl.conf file must be included in your AxoSyslog configuration:
@include "scl.conf"
The panos-parser() is a reusable configuration snippet configured to parse Palo Alto Networks PAN-OS log messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
13.18.1 - Message format parsed by panos-parser()
This section illustrates the most commonly used PAN-OS log format on the AxoSyslog side.
There are several “types” of log formats in Palo Alto Networks PAN-OS. For example, the most commonly used SYSTEM type has the following message format on the AxoSyslog side after parsing:
<12>Apr 14 16:48:54 paloalto.test.net 1,2020/04/14 16:48:54,unknown,SYSTEM,auth,0,2020/04/14 16:48:54,,auth-fail,,0,0,general,medium,failed authentication for user 'admin'. Reason: Invalid username/password. From: 10.0.10.55.,1718,0x0,0,0,0,0,,paloalto
13.18.2 - PAN-OS parser options
The panos-parser() has the following options:
prefix()
Synopsis:
prefix()
Default:
“.panos.”
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
13.19 - PostgreSQL csvlog
Available in AxoSyslog version 4.5.0 and later.
This parser processes messages in the PostgreSQL csvlog format.
The following sample message is a multi-line message with embedded NL characters. This is a single, multi-line log entry that starts with the timestamp.
2023-08-08 12:05:52.805 UTC,,,22113,,64d22fa0.5661,1,,2023-08-08 12:05:52 UTC,23/74060,0,LOG,00000,"automatic vacuum of table ""tablename"": index scans: 0
pages: 0 removed, 4 remain, 0 skipped due to pins, 0 skipped frozen
tuples: 114 removed, 268 remain, 0 are dead but not yet removable, oldest xmin: 149738000
buffer usage: 97 hits, 0 misses, 6 dirtied
avg read rate: 0.000 MB/s, avg write rate: 114.609 MB/s
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s",,,,,,,,,""
The postgresql-csvlog-parser() extracts the information from this message into a set of name-value pairs. By default, the name-value pairs have the .pgsql prefix.
The postgresql-csvlog-parser() driver is actually a reusable configuration snippet configured to parse log messages using the csv-parser(). For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
on-type-error()
Synopsis:
string
Description: Specifies what to do when casting a parsed value to a specific data type fails. Note that the flags(drop-invalid) option and the on-error() global option also affects the behavior.
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
13.20 - Python parser
The Python log parser (available in AxoSyslog version 3.10 and later) allows you to write your own parser in Python. Practically, that way you can process the log message (or parts of the log message) any way you need. For example, you can import external Python modules to process the messages, query databases to enrich the messages with additional data, and many other things.
The following points apply to using Python blocks in AxoSyslog in general:
Python parsers and template functions are available in AxoSyslog version 3.10 and later.
Python destinations and sources are available in AxoSyslog version 3.18 and later.
Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).
The Python block must be a top-level block in the AxoSyslog configuration file.
If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.
In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".
The Python object is initiated every time when AxoSyslog is started or reloaded.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
The Python block can contain multiple Python functions.
Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.
Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.
Python error messages are available in the internal() source of AxoSyslog.
You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.
Declaration:
Python parsers consist of two parts. The first is a AxoSyslog parser object that you use in your AxoSyslog configuration, for example, in the log path. This parser references a Python class, which is the second part of the Python parsers. The Python class processes the log messages it receives, and can do virtually anything that you can code in Python.
parser <name_of_the_python_parser>{ python( class("<name_of_the_python_class_executed_by_the_parser>"));}; python { class MyParser(object):
def init(self, options):
'''Optional. This method is executed when syslog-ng is started or reloaded.'''return True
def deinit(self):
'''Optional. This method is executed when syslog-ng is stopped or reloaded.''' pass
def parse(self, msg):
'''Required. This method receives and processes the log message.'''return True
};
Methods of the python() parser
The init (self, options) method (optional)
The AxoSyslog application initializes Python objects only when it is started or reloaded. That means it keeps the state of internal variables while AxoSyslog is running. The init method is executed as part of the initialization. You can perform any initialization steps that are necessary for your parser to work. For example, if you want to perform a lookup from a file or a database, you can open the file or connect to the database here, or you can initialize a counter that you will increase in the parse() method.
The return value of the init() method must be True. If it returns False, or raises an exception, AxoSyslog will not start.
options: This optional argument contains the contents of the options() parameter of the parser object as a Python dict.
The parse() method processes the log messages it receives, and can do virtually anything that you can code in Python. This method is required, otherwise AxoSyslog will not start.
The return value of the parse() method must be True. If it returns False, or raises an exception, AxoSyslog will drop the message.
To reference a name-value pair or a macro in the Python code, use the following format. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message['HOST'], and so on. (The log-message contains the entire log message (not just the text body) in a structure similar to a Python dict, but it is actually an object.)
You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.
Note that the names of the name-value pairs are case-sensitive. If you create a new name-value pair called new-macro-name in Python, and want to reference it in another part of the AxoSyslog configuration file (for example, in a template), use the ${new-macro-name} macro.
To list all available keys (names of name-value pairs), use the log_message.keys() function.
The deinit(self) method (optional)
This method is executed when AxoSyslog is stopped or reloaded.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
Example: Parse loggen logs
The following sample code parses the messages of the loggen tool (for details, see The loggen manual page). The following is a sample loggen message:
The AxoSyslog parser object references the LoggenParser class and passes a set of regular expressions to parse the loggen messages. The init() method of the LoggenParser class compiles these expressions into a pattern. The parse method uses these patterns to extract the fields of the message into name-value pairs. The destination template of the AxoSyslog log statement uses the extracted fields to format the output message.
Example: Parse Windows eventlogs in Python - performance
The following example uses regular expressions to process Windows log messages received in XML format. The parser extracts different fields from messages received from the Security and the Application eventlog containers. Using the following configuration file, AxoSyslog could process about 25000 real-life Windows log messages per second.
@version: 4.5.0
options { keep-hostname(yes); keep-timestamp(no); stats-level(2); use-dns(no);};source s_network_aa5fdf25c39d4017a8e504cdb641b477 { network( flags(no-parse) ip(0.0.0.0) log-fetch-limit(1000) log-iw-size(100000) max-connections(100) port(514));}; parser p_python_parser_79c31da44bb64de6b5de84be4ae15a15 { python(options("regex_for_security", ".* Security ID: (?P<security_id>\\S+) Account Name: (?P<account_name>\\S+) Account Domain: (?P<account_domain>\\S+) Logon ID: (?P<logon_id>\\S+).*Process Name: (?P<process_name>\\S+).*EventID (?P<event_id>\\d+)", "regex_others", "(.*)EventID (?P<event_id>\\d+)") class("EventlogParser"));}; destination d_file_78363e1dd90c4ebcbb0ee1eff5a2e310 { file("/var/testdb_working_dir/fcd713a2-d48e-4025-9192-ec4a9852cafa.$HOST" flush-lines(1000) log-fifo-size(200000));}; log { source(s_network_aa5fdf25c39d4017a8e504cdb641b477); parser(p_python_parser_79c31da44bb64de6b5de84be4ae15a15); destination(d_file_78363e1dd90c4ebcbb0ee1eff5a2e310); flags(flow-control);}; python { import re
class EventlogParser(object):
def init(self, options):
self.regex_security = re.compile(options["regex_for_security"]) self.regex_others = re.compile(options["regex_others"])return True
def deinit(self):
pass
def parse(self, log_message):
security_match= self.regex_security.match(log_message['MESSAGE'])if security_match:
for key, value in security_match.groupdict().items():
log_message[key]= value
else:
others_match= self.regex_others.match(log_message['MESSAGE'])if others_match:
for key, value in others_match.groupdict().items():
log_message[key]= value
return True
};
13.21 - Regular expression (regexp) parser
The AxoSyslog application can parse fields from a message with the help of regular expressions. This can be also achieved with the match() filter, by setting the store-matches flag, but the regexp-parser() offers more flexibility, like multiple patterns and setting the prefix of the created name-value pairs.
Note
The regexp-parser() can create additional name-value pairs only if “named capture groups” are used in the regular expression, for example (?<test_field>\w+). For more information, see “named capture groups” in PCRE documentation.
For more information about regular expressions in AxoSyslog, see Regular expressions.
In the following example, the incoming log message is the following:
Apr 20 11:09:46 test_field -> test_value
The regexp-parser inserts the .regexp. prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (.) character will be written to the file (dot-nv-pairs). The log line connects the source, the parser and the destination.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
patterns()
Synopsis:
patterns(“pattern1” “pattern2”)
Mandatory:
yes
Description: The regular expression patterns that you want to find a match. regexp-parser() supports multiple patterns, and stops the processing at the first successful match.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
This parser does not have a default prefix. To configure a custom prefix, use the following format:
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
13.22 - Structured data (SDATA) parser
Available in AxoSyslog 4.1 and later.
The sdata-parser() allows you to parse an RFC5424-style structured data string. You can use it to parse this relatively complex format separately, for example, to process malformatted messages. You can use the optional prefix option to add a specific string before the names of the parsed name-value pairs.
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
template()
Synopsis:
template("${<macroname>}")
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
13.23 - Sudo parser
The sudo parser can parse the log messages of the sudo command. Available in version 3.16 and later.
The sudo-parser() is actually a reusable configuration snippet configured to parse sudo messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, sudo-parser() uses the .sudo. prefix. To modify it, use the following format:
parser { sudo-parser(prefix("myprefix."));};
13.24 - Parsing syslog messages
By default, AxoSyslog parses every message using the syslog-parser as a syslog message, and fills the macros with values of the message. The syslog-parser does not discard messages: the message cannot be parsed as a syslog message, the entire message (including its header) is stored in the ${MSG} macro. If you do not want to parse the message as a syslog message, use the flags(no-parse) option of the source.
You can also use the syslog-parser to explicitly parse a message, or a part of a message as a syslog message (for example, after rewriting the beginning of a message that does not comply with the syslog standards).
Example: Using junctions
For example, suppose that you have a single network source that receives log messages from different devices, and some devices send messages that are not RFC-compliant (some routers are notorious for that). To solve this problem in earlier versions of AxoSyslog, you had to create two different network sources using different IP addresses or ports: one that received the RFC-compliant messages, and one that received the improperly formatted messages (for example, using the flags(no-parse) option). Using junctions this becomes much more simple: you can use a single network source to receive every message, then use a junction and two channels. The first channel processes the RFC-compliant messages, the second everything else. At the end, every message is stored in a single file. The filters used in the example can be host() filters (if you have a list of the IP addresses of the devices sending non-compliant messages), but that depends on your environment.
Since every channel receives every message that reaches the junction, use the flags(final) option in the channels to avoid the unnecessary processing the messages multiple times:
Note that by default, the syslog-parser attempts to parse the message as an RFC3164-formatted (BSD-syslog) message. To parse the message as an RFC5424-formatted message, use the flags(syslog-protocol) option in the parser.
syslog-parser(flags(syslog-protocol));
Parsing errors
AxoSyslog 4.7 and newer automatically adds the following tags if it encounters errors when parsing syslog messages.
message.utf8_sanitized
message.parse_error
syslog.missing_pri
syslog.missing_timestamp
syslog.invalid_hostname
syslog.unexpected_framing
syslog.rfc3164_missing_header
syslog.rfc5424_unquoted_sdata_value
13.24.1 - Options of syslog-parser() parsers
The syslog-parser() has the following options:
default-facility()
Type:
facility string
Default:
kern
Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.
default-priority()
Type:
priority string
Default:
Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).
drop-invalid()
Type:
yes or no
Values:
`yes
Default:
no
Description: This option determines how the syslog-parser() affects messages when parsing fails.
If you set drop-invalid() to yes, syslog-parser() will drop the message if the parsing fails.
If you set drop-invalid() to no, the parsing error triggers syslog-parser() to rewrite and extend the original log message with the following additional information:
It prepends the following message to the contents of the $MESSAGE field: Error processing log message.
It sets the contents of the $PROGRAM field to syslog-ng.
It sets the contents of the facility field to syslog.
It sets the contents of the severity field to error.
Note
With the drop-invalid(no) option syslog-parser() will work in the same way as the sources which receive syslog-protocol/BSD-format messages.
Description: Specifies the log parsing options of the source.
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
Example: using the no-header flag with the syslog-parser() parser
The following example illustrates using the no-header flag with the syslog-parser() parser:
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
sdata-prefix()
Type:
string
Default:
.SDATA.
Available in AxoSyslog 4.1 and later.
Description: Adds a specific string before the names of the parsed SDATA fields to store the name-value pairs created from the SDATA fields separately. Note that unless the value of sdata-prefix starts with .SDATA., using this option excludes the parsed fields from the sdata and rfc5424 scopes of the value pairs.
template()
Synopsis:
template("${<macroname>}")
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
13.25 - Parsing tags
The AxoSyslog application can tag log messages, and can include these tags in the log messages, as described in Tagging messages. The tags-parser() can parse these tags from the incoming messages and re-tag them. That way if you add tags to a log message on a AxoSyslog client, the message will have the same tags on the AxoSyslog server. Available in version 3.23 and later.
Specify the macro that contains the list of tags to parse in the template() option of the parser, for example, the SDATA field that you used to transfer the tags, or the name of the JSON field that contains the tags after using the json-parser().
The Websense parser can parse the log messages of Websense Content Gateway (Raytheon|Websense, now Forcepoint). These messages do not completely comply with the syslog RFCs, making them difficult to parse. The websense-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:
If you find a message that the websense-parser() cannot properly parse, contact us, so we can improve the parser.
The AxoSyslog application sets the ${PROGRAM} field to Websense.
By default, the websense-specific fields are extracted into name-value pairs prefixed with .websense. For example, the product_version in the previous message becomes ${.websense.product_version}. You can change the prefix using the prefix option of the parser.
Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The websense-parser() is actually a reusable configuration snippet configured to parse websense messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, websense-parser() uses the .websense. prefix. To modify it, use the following format:
parser { websense-parser(prefix("myprefix."));};
13.27 - Windows XML Event Log (EVTX) parser
Available in AxoSyslog version 4.5 and later.
The new windows-eventlog-xml-parser() can parse messages in the Windows XML Event Log (EVTX) format.
Extensible Markup Language (XML) is a text-based open standard designed for both human-readable and machine-readable data interchange. Like JSON, it is used primarily to transmit data between a server and web application. It is described in W3C Recommendation: Extensible Markup Language (XML).
The XML parser processes input in XML format, and adds the parsed data to the message object.
To create an XML parser, define an xml_parser that has the xml() option. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message using the XML parser, use the template() option. You can also define the parser inline in the log path.
In the following example, the source is an XML-encoded log message. The destination is a file that uses the format-json template. The log line connects the source, the destination and the parser.
The XML parser inserts an “.xml” prefix by default before the extracted name-value pairs. Since format-json replaces a dot with an underscore at the beginning of keys, the “.xml” prefix becomes “_xml”. Attributes get an _ prefix. For example, from the XML input:
When the text is separated by tags on different levels or tags on the same level, the parser simply concatenates the different parts of text. For example, from this input XML:
However, note that users can choose to strip whitespaces using the strip-whitespaces() option.
Configuration hints
Define a source that correctly detects the end of the message, otherwise the XML parser will consider the input invalid, resulting in a parser error.
To ensure that the end of the XML document is accurately detected, use any of the following options:
Ensure that the XML is a single-line message.
In the case of multiline XML documents:
If the opening and closing tags are fixed and known, you can use multi-line-mode(prefix-suffix). Using regular expressions, specify a prefix and suffix matching the opening and closing tags. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.
In the case of TCP, you can encapsulate and send the document in syslog-protocol format, and use a syslog() source. Make sure that the message conforms to the octet counting method described in RFC6587.
Considering the new lines as one character, 59 is appended to the original message.
You can use a datagram-based source. In the case of datagram-based sources, the protocol signals the end of the message automatically. Ensure that the complete XML document is written in one message.
Unless the opening and closing tags are fixed and known, stream-based sources are currently not supported.
In case you experience issues, start syslog-ng with debug logs enabled. There will be a debug log about the incoming log entry, which shows the complete message to be parsed. The entry should contain the entire XML document.
Note
If your log messages are entirely in .xml format, make sure to disable any message parsing on the source side by including the flags("no-parse") option in your source statement. This will put the entire log message in the $MESSAGE macro, which is the field that the XML parser parses by default.
13.28.1 - Limitations of the XML parsers
The XML parser comes with certain limitations.
Vector-like structures:
It is not possible to address each element of a vector-like structure individually. For example, take this input:
The XML parser does not support CDATA. CDATA inside the XML input is ignored. This is true for the processing instructions as well.
Inherited limitations:
The XML parser is based on the glib XML subset parser, called “GMarkup” parser, which is not a full-scale XML parser. It is intended to parse a simple markup format that is a subset of XML. Some limitations are inherited:
Do not use the XML parser if you expect to interoperate with applications generating full-scale XML. Instead, use it for application data files, configuration files, log files, and so on, where you know your application will be the only one writing the file.
The XML parser is not guaranteed to display an error message in the case of invalid XML. It may accept invalid XML. However, it does not accept XML input that is not well-formed (a condition that is weaker than requiring XML to be valid).
No support for long keys:
If the key is longer than 255 characters, AxoSyslog drops the entry and an error log is emitted. There is no chunking or any other way of recovering data, not even partial data. The entry will be replaced by an empty string.
13.28.2 - Options of the XML parsers
The XML parser has the following options.
drop-invalid
Synopsis:
drop-invalid()
Format:
yes
Default:
no
Mandatory:
no
Description: If set, messages with an invalid XML will be dropped entirely.
exclude-tags
Synopsis:
exclude-tags()
Format:
list of globs
Default:
None
If not set, no filtering is done.
Mandatory:
no
Description: The XML parser matches tags against the listed globs. If there is a match, the given subtree of the XML will be omitted.
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
The prefix() option is optional and its default value is ".xml".
strip-whitespaces
Synopsis:
strip-whitespaces()
Format:
yes
Default:
no
Mandatory:
no
Description: Strip the whitespaces from the XML text nodes before adding them to the message.
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
14 - Correlating log messages
The AxoSyslog application can correlate log messages. Alternatively, you can also correlate log messages using pattern databases. For details, see Correlating log messages using pattern databases.
To group or correlate log messages that match a set of filters, use the group-by parser. This works similarly to SQL GROUP BY statements. For details, see Correlating messages using the grouping-by() parser.
Log messages are supposed to describe events, but applications often separate information about a single event into different log messages. For example, the Postfix email server logs the sender and recipient addresses into separate log messages, or in case of an unsuccessful login attempt, the OpenSSH server sends a log message about the authentication failure, and the reason of the failure in the next message. Of course, messages that are not so directly related can be correlated as well, for example, login-logout messages, and so on.
To correlate log messages with AxoSyslog, you can add messages into message-groups called contexts. A context consists of a series of log messages that are related to each other in some way, for example, the log messages of an SSH session can belong to the same context. As new messages come in, they may be added to a context. Also, when an incoming message is identified it can trigger actions to be performed, for example, generate a new message that contains all the important information that was stored previously in the context.
14.1 - Correlating messages using the grouping-by() parser
The AxoSyslog application can correlate log messages that match a set of filters. This works similarly to SQL GROUP BY statements. Alternatively, you can also correlate log messages using pattern databases. For details, see Correlating log messages using pattern databases.
Log messages are supposed to describe events, but applications often separate information about a single event into different log messages. For example, the Postfix email server logs the sender and recipient addresses into separate log messages, or in case of an unsuccessful login attempt, the OpenSSH server sends a log message about the authentication failure, and the reason of the failure in the next message. Of course, messages that are not so directly related can be correlated as well, for example, login-logout messages, and so on.
To correlate log messages with AxoSyslog, you can add messages into message-groups called contexts. A context consists of a series of log messages that are related to each other in some way, for example, the log messages of an SSH session can belong to the same context. As new messages come in, they may be added to a context. Also, when an incoming message is identified it can trigger actions to be performed, for example, generate a new message that contains all the important information that was stored previously in the context.
How the grouping-by() parser works
The grouping-by() parser has three options that determine if a message is added to a context: scope(), key(), and where().
The scope() option acts as an early filter, selecting messages sent by the same process (${HOST}${PROGRAM}${PID} is identical), application (${HOST}${PROGRAM} is identical), or host.
The key() identifies the context the message belongs to. (The value of the key must be the same for every message of the context.)
To use a filter to further limit the messages that are added to the context, you can use the where() option.
The timeout() option determines how long a context is stored, that is, how long AxoSyslog waits for related messages to arrive. If the group has a specific log message that ends the context (for example, a logout message), you can specify it using the trigger() option.
When the context is closed, and the messages match the filter set in the having() option (or the having() option is not set), AxoSyslog generates and sends the message set in the aggregate() option.
Note
Message contexts are persistent and are not lost when AxoSyslog is reloaded (SIGHUP), but are lost when AxoSyslog is restarted.
For the parser to work, you must set at least the following options: key(), aggregate(), and timeout().
Note the following points about timeout values:
When a new message is added to a context, AxoSyslog will restart the timeout using the context-timeout set for the new message.
When calculating if the timeout has already expired or not, AxoSyslog uses the timestamps of the incoming messages, not system time elapsed between receiving the two messages (unless the messages do not include a timestamp, or the keep-timestamp(no) option is set). That way AxoSyslog can be used to process and correlate already existing log messages offline. However, the timestamps of the messages must be in chronological order (that is, a new message cannot be older than the one already processed), and if a message is newer than the current system time (that is, it seems to be coming from the future), AxoSyslog will replace its timestamp with the current system time.
If the context-timeout is 10 seconds and AxoSyslog receives the messages within 1 second, the timeout event will occour immediately, because the difference of the two timestamp (60 seconds) is larger than the timeout value (10 seconds).
Avoid using unnecessarily long timeout values on high-traffic systems, as storing the contexts for many messages can require considerable memory. For example, if two related messages usually arrive within seconds, it is not needed to set the timeout to several hours.
Example: Correlating Linux Audit logs
Linux audit logs tend to be broken into several log messages (generated as a list of lines). Usually, the related lines are close to each other in time, but multiple events can be logged at around the same time, which get mixed up in the output. The example below is the audit log for running ntpdate:
These lines are connected by their second field: msg=audit(1440927434.124:40347). You can parse such messages using the Linux audit parser of AxoSyslog, and then use the parsed .auditd.msg field to group the messages.
14.1.1 - Referencing earlier messages of the context
When creating the aggregated message, or in the various parameters of the grouping-by() parser, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For example, if there are three log messages in a context, the ${HOST}@1 expression refers to the host field of the current (third) message in the context, the ${HOST}@2 expression refers to the host field of the previous (second) message in the context, ${PID}@3 to the PID of the first message, and so on. For example, the following message can be created from SSH login/logout messages: An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}.
Warning
When referencing an earlier message of the context, always enclose the field name between braces, for example, ${PID}@3. The reference will not work if you omit the braces.
Note
To use a literal @ character in a template, use @@.
Example: Referencing values from an earlier message
The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):
aggregate( value('value name="MESSAGE" An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}'))
If you do not know in which message of the context contains the information you need, you can use the grep template function. For details, see grep.
Example: Using the grep template function
The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.
$(grep ("${username}"=="root")${auth_method})
To perform calculations on fields that have numerical values, see Numerical operations.
14.1.2 - Options of grouping-by parsers
The grouping-by has the following options.
aggregate()
Synopsis:
aggregate()
Description: Specifies the message that AxoSyslog generates when the context is closed. This option is mandatory.
Note that the aggregate() option has access to every message of the context, and has the following options:
inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.
context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.
last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.
none: An empty message is created, without inheriting any tags or name-value pairs.
value: Adds a name-value pair to the generated message. You can include text, macros, template functions, and you can also reference every message of the context. For details on accessing other messages of the context, see Referencing earlier messages of the context.
having()
Synopsis:
having()
Description: Specifies a filter: AxoSyslog generates the aggregate message only if the result of the filter expression is true. Note that the having() filter has access to every message of the context. For details on accessing other messages of the context, see Referencing earlier messages of the context.
inject-mode()
Synopsis:
inject-mode()
Description: By default, the aggregated message that AxoSyslog generates is injected into the same place where the grouping-by() statement is referenced in the log path. To post the generated message into the internal() source instead, use the inject-mode() option in the definition of the parser.
Example: Sending triggered messages to the internal() source
To send the generated messages to the internal source, use the inject-mode("internal") option:
You can configure the generated message in the aggregate() option (see aggregate()). You can create an entire message, use macros and values extracted from the original message, and so on.
key()
Synopsis:
key()
Description: Specifies the key as a template (that is, the name of a name-value pair) that every message must have to be added to the context. The value of the key must be the same for every message of the context. For example, this can be a session-id parsed from firewall messages, and so on.
This is a mandatory option.
Note
Messages that do not have a key will all belong to the same context.
Note
If the value of the key is static (for example, key("PROGRAM") instead of key("$PROGRAM")), all messages will belong to the same context.
scope()
Type:
process, program, host, or global
Default:
global
Description: Specifies which messages belong to the same context. The following values are available:
process: Only messages that are generated by the same process of a client belong to the same context, that is, messages that have identical ${HOST}, ${PROGRAM} and ${PID} values.
program: Messages that are generated by the same application of a client belong to the same context, that is, messages that have identical ${HOST} and ${PROGRAM} values.
host: Every message generated by a client belongs to the same context, only the ${HOST} value of the messages must be identical.
global: Every message belongs to the same context. This is the default value.
sort-key()
Synopsis:
sort-key()
Description: Allows sorting of the elements before they are aggregated into a context. Use this when entries are not received in order. This works similarly to the SQL ORDER BY keyword.
Note
Sorting is done by AxoSyslog when the context is about to be closed by trigger() or timeout(), but before AxoSyslog evaluates the having() option.
AxoSyslog can slow down if you specify several sort-key macro or template options, for example, sort-key("${3}${4}").
timeout()
Synopsis:
timeout([seconds])
Description: Specifies the maximum time to wait for all messages of the context to arrive. If no new message is added to the context during this period, the context is assumed to be complete and AxoSyslog generates and sends the triggered message (specified in the aggregate() option), and clears the context. If a new message is added to the context, the timeout period is restarted.
This option is mandatory, and its value must be equal to or greater than 1.
trigger()
Synopsis:
trigger()
Description: A filter that specifies the final message of the context. If the filter matches the incoming message, AxoSyslog generates and sends the triggered message (specified in the aggregate() option), and clears the context.
where()
Synopsis:
where()
Description: Specifies a filter condition. Messages not matching the filter will not be added to the context. Note that the where() filter has access only to the current message.
15 - Enriching log messages with external data
To properly interpret the events that the log messages describe, you must be able to handle log messages as part of a system of events, instead of individual information chunks. The AxoSyslog application allows you to import data from external sources to include in the log messages, thus extending, enriching, and complementing the data found in the log message.
The AxoSyslog application currently provides the following possibilities to enrich log messages.
You can write custom Python modules to process the messages and add data from external files or databases. For details, see Python parser.
15.1 - Adding metadata from an external file
In AxoSyslog version 3.8 and later, you can use an external database file to add additional metadata to your log messages. For example, you can create a database (or export it from an existing tool) that contains a list of hostnames or IP addresses, and the department of your organization that the host belongs to, the role of the host (mailserver, webserver, and so on), or similar contextual information.
The database file is a simple text file in comma-separated value (CSV) format, where each line contains the following information:
A selector or ID that appears in the log messages, for example, the hostname. To use shell-style globbing (wildcards) in selectors, see Shell-style globbing in the selector. You can also reference the name of a filter that matches the messages, see Using filters as selector
The name of the name-value pair that AxoSyslog adds to matching log messages.
The value of the name-value pairs. Starting with AxoSyslog version 3.22, the value of the name-value pair can be a template or a template function, for example, "selector3,name,$(echo $HOST_FROM)";
For example, the following csv-file contains three lines identified with the IP address, and adds the host-role field to the log message.
You can also add data to messages that do not have a matching selector entry in the database using the default-selector() option.
If you modify the database file, you have to reload AxoSyslog for the changes to take effect. If reloading AxoSyslog or the database file fails for some reason, AxoSyslog will keep using the last working database file.
Example: Adding metadata from a CSV file
The following example defines uses a CSV database to add the role of the host based on its IP address, and prefixes the added name-value pairs with .metadata. The destination includes a template that simply appends the added name-value pairs to the end of the log message.
To better control to which log messages you add contextual data, you can use filters as selectors. In this case, the first column of the CSV database file must contain the name of a filter. For each message, AxoSyslog evaluates the filters in the order they appear in the database file. If a filter matches the message, AxoSyslog adds the name-value pair related to the filter.
For example, the database file can contain the entries. (For details on the accepted CSV-format, see database().)
Note that AxoSyslog does not evaluate other filters after the first match. For example, if you use the previous database file, and a message matches both the f_auth and f_localhost filters, AxoSyslog adds only the name-value pair of f_auth to the message.
To add multiple name-value pairs to a message, include a separate line in the database for each name-value pair, for example:
You can also add data to messages that do not have a matching selector entry in the database using the default-selector() option.
You must store the filters you reference in a database in a separate file. This file is similar to a AxoSyslog configuration file, but must contain only a version string and filters (and optionally comments). You can use the `syslog-ng –syntax-only command to ensure that the file is valid. For example, the content of such a file can be:
If you modify the database file, or the file that contains the filters, you have to reload AxoSyslog for the changes to take effect. If reloading AxoSyslog or the files fails for some reason, AxoSyslog will keep using the last working version of the file.
15.1.2 - Shell-style globbing in the selector
Starting with in AxoSyslog 3.24 and later, you can use shell-style globbing (’*’ and ‘?’ wildcards) in the selector.
To use globs in a selector
Use the glob() option within the selector() option in your AxoSyslog configuration file, for example:
The add-contextual-data() has the following options.
Required options:
The following options are required: selector(), database().
database()
Type:
.csv
Default:
Description: Specifies the path to the CSV file, for example, /opt/syslog-ng/my-csv-database.csv. The extension of the file must be .csv, and can include Windows-style (CRLF) or UNIX-style (LF) linebreaks. You can use absolute path, or relative to the syslog-ng binary.
default-selector()
Synopsis:
default-selector()
Description: Specifies the ID of the entry (line) that is corresponds to log messages that do not have a selector that matches an entry in the database. For example, if you add name-value pairs from the database based on the hostname from the log message (selector("${HOST}")), then you can include a line for unknown hosts in the database, and set default-selector() to the ID of the line for unknown hosts. In the CSV file:
Description: Specifies if selectors are handled as case insensitive. If you set the ignore-case() option to yes, selectors are handled as case insensitive.
prefix()
Synopsis:
prefix()
Description: Insert a prefix before the name part of the added name-value pairs (including the pairs added by the default-selector()) to help further processing.
selector()
Synopsis:
selector()
Description: Specifies the string or macro that AxoSyslog evaluates for each message, and if its value matches the ID of an entry in the database, AxoSyslog adds the name-value pair of every matching database entry to the log message. You can use the following in the selector() option.
The AxoSyslog application can lookup IPv4 addresses from an offline GeoIP database, and make the retrieved data available in name-value pairs. IPv6 addresses are not supported. Depending on the database used, you can access country code, longitude, and latitude information.
Note
To access longitude and latitude information, download the GeoLiteCity database, and unzip it (for example, to the /usr/share/GeoIP/GeoLiteCity.dat file). The default databases available on Linux and other platforms usually contain only the country codes.
You can refer to the separated parts of the message using the key of the value as a macro. For example, if the message contains KEY1=value1,KEY2=value2, you can refer to the values as ${KEY1} and ${KEY2}.
In the following example, AxoSyslog retrieves the GeoIP data of the IP address contained in the ${HOST} field of the incoming message, and includes the data (prefixed with the geoip. string) in the output JSON message.
If you are transferring your log messages into Elasticsearch, use the following rewrite rule to combine the longitude and latitude information into a single value (called geoip.location), and set the mapping in Elasticsearch accordingly. Do not forget to include the rewrite in your log path. For details on transferring your log messages to Elasticsearch, see elasticsearch2: DEPRECATED - Send messages directly to Elasticsearch version 2.0 or higher.
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
For example, to insert the geoip. prefix, use the prefix(.geoip.) option. To refer to a particular data when using a prefix, use the prefix in the name of the macro, for example, ${geoip.country_code} .
database()
Synopsis:
database()
Default:
/usr/share/GeoIP/GeoIP.dat
Description: The full path to the GeoIP database to use. Note that AxoSyslog must have the required privileges to read this file. Do not modify or delete this file while AxoSyslog is running, it can crash AxoSyslog.
15.3 - Looking up GeoIP2 data from IP addresses
The AxoSyslog application can lookup IP addresses from an offline GeoIP2 database, and make the retrieved data available in name-value pairs. Depending on the database used, you can access country code, longitude, and latitude information and so on.
The AxoSyslog application works with the Country and the City version of the GeoIP2 database, both free and the commercial editions. The AxoSyslog application works with the mmdb (GeoIP2) format of these databases. Other formats, like csv are not supported.
Note
To access longitude and latitude information, download the City version of the GeoIP2 database.
There are two types of GeoIP2 databases available.
GeoLite2 City:
free of charge
less accurate
GeoIP2 City:
has to be purchased
more accurate
Unzip the downloaded database (for example, to the /usr/share/GeoIP2/GeoIP2City.mmdb file). This path will be used later in the configuration.
Starting with version 3.24, AxoSyslog tries to automatically detect the location of the database. If that is successful, the database() option is not mandatory.
15.3.1 - Referring to parts of the message as a macro
You can refer to the separated parts of the message using the key of the value as a macro. For example, if the message contains KEY1=value1,KEY2=value2, you can refer to the values as ${KEY1} and ${KEY2}.
for example, if the default prefix (.geoip2) is used, you can determine the country code using ${.geoip2.country.iso_code}.
To look up all keys:
Install the mmdb-bin package.
After installing this package, you will be able to use the mmdblookup command.
Note
The name of the package depends on the Linux distribution. The package mentioned in this example is on Ubuntu.
Create a dump using the following command: `mmdblookup –file GeoLite2-City.mmdb –ip
The resulting dump file will contain the keys that you can use.
For a more complete list of keys, you can also check the GeoIP2 City and Country CSV Databases. However, note that the AxoSyslog application works with the mmdb (GeoIP2) format of these databases. Other formats, like csv are not supported.
In the following example, AxoSyslog retrieves the GeoIP2 data of the IP address contained in the ${HOST} field of the incoming message (assuming that in this case the ${HOST} field contains an IP address), and includes the data (prefixed with the geoip2 string) in the output JSON message.
15.3.3 - Transferring your logs to Elasticsearch using GeoIP2
If you are transferring your log messages into Elasticsearch, use the following rewrite rule to combine the longitude and latitude information into a single value (called geoip2.location), and set the mapping in Elasticsearch accordingly. Do not forget to include the rewrite in your log path. These examples assume that you used prefix("geoip2.") instead of the default for the geoip2 parser. For details on transferring your log messages to Elasticsearch, see elasticsearch2: DEPRECATED - Send messages directly to Elasticsearch version 2.0 or higher.
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
For example, to insert the .geoip2 prefix, use the prefix(.geoip2) option. To refer to a particular data when using a prefix, use the prefix in the name of the macro, for example, ${geoip2.country_code} .
database()
Synopsis:
database()
Default:
Description: Path to the GeoIP2 database to use. This works with absolute and relative paths as well. Note that AxoSyslog must have the required privileges to read this file. Do not modify or delete this file while AxoSyslog is running, it can crash AxoSyslog.
Starting with version 3.24, AxoSyslog tries to automatically detect the location of the database. If that is successful, the database() option is not mandatory.
16 - Statistics of AxoSyslog
The AxoSyslog application collects various statistics and measures different metrics about the messages it receives and delivers. These metrics are collected into different counters, depending on the configuration of AxoSyslog. The stats-level() global option determines exactly which statistics AxoSyslog collects. You can access these statistics and metrics using the following methods.
Use the socat application: echo STATS | socat -vv UNIX-CONNECT:/opt/syslog-ng/var/run/syslog-ng.ctl -
If you have an OpenBSD-style netcat application installed, use the echo STATS | nc -U /opt/syslog-ng/var/run/syslog-ng.ctl command. Note that the netcat included in most Linux distributions is a GNU-style version that is not suitable to query the statistics of syslog-ng.
16.1 - Metrics and counters
You can list all active metrics on your AxoSyslog host using the following command (this lists the metrics, without their current values): syslog-ng-ctl query list "*"
To list the metrics and their values, use the following command: syslog-ng-ctl query get "*"
The displayed metrics have the following structure.
The type of the object (for example, dst.file, tag, src.facility)
The ID of the object used in the syslog-ng.conf configuration file, for example, d_internal or source.src_tcp. The #0 part means that this is the first destination in the destination group.
The instance ID (destination) of the object, for example, the filename of a file destination, or the name of the application for a program source or destination.
The status of the object. One of the following:
a - active. At the time of querying the statistics, the source or the destination was still alive (it continuously received statistical data).
d - dynamic. Such objects may not be continuously available, for example, like statistics based on the sender’s hostname. These counters only appear above a certain value of stats-level() global option:
host: source host, from stats-level(2)
program: program, from stats-level(3)
sender: sender host, from stats-level(3)
Example: Dynamic counters
The following example contains 6 different dynamic values: a sender, a host, and four different programs.
To avoid performance issues or even overloading AxoSyslog, you might want to limit the number of registered dynamic counters in the message statistics. To do this, configure the stats-max-dynamics() global option.
o - This object was once active, but stopped receiving messages. (For example, a dynamic object may disappear and become orphan.)
Note
The AxoSyslog application stores the statistics of the objects when AxoSyslog is reloaded. However, if the configuration of AxoSyslog was changed since the last reload, the statistics of orphaned objects are deleted.
The connections statistics counter displays the number of connections tracked by AxoSyslog for the selected source driver.
Example: sample configuration and statistics output
The following configuration will display the following syslog-ng-ctl statistics output:
batch_size_avg: When batching is enabled, then this shows the current average batch size of the given source or destination.
batch_size_max: When batching is enabled, the value of batch_size_max shows the current largest batch size of the given source or destination.
discarded: The number of messages discarded by the given parser. These are messages that the parser could not parsed, and are therefore not processed. For example:
parser;demo_parser;;a;discarded;20
dropped: The number of dropped messages — AxoSyslog could not send the messages to the destination and the output buffer got full, so messages were dropped by the destination driver, or AxoSyslog dropped the message for some other reason (for example, a parsing error).
eps_last_1h: The EPS value of the past 1 hour.
eps_last_24h: The EPS value of the past 24 hours.
eps_since_start: The EPS value since the current AxoSyslog start.
Note
When using the eps_last_1h, the eps_last_24h, and the eps_since_start statistics, consider the following:
EPS stands for “event per second”, and in our case, a message received or sent counts as a single event.
The eps_last_1h, the eps_last_24h, and the eps_since_start values are only approximate values.
The eps_last_1h, the eps_last_24h, and the eps_since_start values are automatically updated every 60 seconds.
matched: The number of messages that are accepted by a given filter. Available for filters and similar objects (for example, a conditional rewrite rule). For example, if a filter matches a specific hostname, then the matched counter contains the number of messages that reached the filter from this hosts.
filter;demo_filter;;a;matched;28
memory_usage: The memory used by the messages in the different queue types (in bytes). This includes every queue used by the object, including memory buffers (log-fifo) and disk-based buffers (both reliable and non-reliable). For example:
The memory usage (size) of queues is not equal to the memory usage (size) of the log messages in AxoSyslog. A log message can be in multiple queues, thus its size is added to multiple queue sizes. To check the size of all log messages, use global.msg_allocated_bytes.value metric.
msg_size_max: The current largest message size of the given source or destination.
msg_size_avg: The current average message size of the given source or destination.
Note
When using the msg_size_avg and msg_size_max statistics, consider that message sizes are calculated as follows:
on the source side: the length of the incoming raw message
on the destination side: the length of the outgoing formatted message
not_matched: The number of messages that are filtered out by a given filter. Available for filters and similar objects (for example, a conditional rewrite rule). For example, if a filter matches a specific hostname, then the not_matched counter contains the number of messages that reached the filter from other hosts, and so the filter discarded them.
Note
Since the not_matched metric applies to filters, and filters are expected to discard messages that do not match the filter condition, not_matched messages are not included in the dropped metric of other objects.
filter;demo_filter;;a;not_matched;0
processed: The number of messages that successfully reached their destination driver.
Note
Consider that a message that has successfully reached its destination driver does not necessarily mean that the destination driver successfully delivered the messages as well. For example, a message can be written to disk or sent to a remote server after reaching the destination driver.
queued: The number of messages passed to the message queue of the destination driver, waiting to be sent to the destination.
stamp: The UNIX timestamp of the last message sent to the destination.
suppressed: The number of suppressed messages (if the suppress() feature is enabled).
written: The number of messages successfully delivered to the destination. This value is calculated from other counters: written = processed - queued - dropped. That is, the number of messages AxoSyslog passed to the destination driver (processed) minus the number of messages that are still in the output queue of the destination driver (queued) and the number of messages dropped because of an error (dropped, for example, because AxoSyslog could not deliver the message to the destination and exceeded the number of retries).
This metric is calculated from other metrics. You cannot reset this metric directly: to reset it, you have to reset the metrics it is calculated from.
Note
Consider that for AxoSyslog version 3.36, the following statistics counters are only supported for the http() destination, or the http() destination and all network() sources and destinations, and all file() sources and destinations, respectively:
Level 0 collects only statistics about the sources and destinations.
Level 1 contains details about the different connections and log files, but has a slight memory overhead.
Level 2 contains detailed statistics based on the hostname.
Level 3 contains detailed statistics based on various message parameters like facility, severity, or tags.
When receiving messages with non-standard facility values (that is, higher than 23), these messages will be listed as other facility instead of their facility number.
Aggregated statistics
Aggregated statistics are available for different sources and destinations from different levels and upwards:
msg_size_avg
msg_size_max
batch_size_avg
batch_size_max
eps_last_1h
eps_last_1h
eps_last_1h
network() source and destination
from level 1
from level 1
counter N/A
counter N/A
from level 1
from level 1
from level 1
file() source and destination
from level 1
from level 1
counter N/A
counter N/A
from level 1
from level 1
from level 1
http() destination
from level 0
from level 0
from level 0
from level 0
from level 0
from level 0
from level 0
16.2 - Log statistics from the internal() source
If the stats-freq() global option is higher than 0, AxoSyslog periodically sends a log statistics message. This message contains statistics about the received messages, and about any lost messages since the last such message. It includes a processed entry for every source and destination, listing the number of messages received or sent, and a dropped entry including the IP address of the server for every destination where AxoSyslog has lost messages. The center(received) entry shows the total number of messages received from every configured sources.
The following is a sample log statistics message for a configuration that has a single source (s_local) and a network and a local file destination (d_network and d_local, respectively). All incoming messages are sent to both destinations.
The statistics include a list of source groups and destinations, as well as the number of processed messages for each. You can control the verbosity of the statistics using the stats-level() global option. The following is an example output.
The statistics are semicolon separated: every line contains statistics for a particular object (for example, source, destination, tag, and so on). The statistics have the following fields:
To reset the statistics to zero, use the following command: syslog-ng-ctl stats --reset
17 - Multithreading and scaling
Starting with version 3.3, AxoSyslog can process sources and destinations in multithreaded mode to scale to multiple CPUs or cores for increased performance. Starting with version 3.6, this multithreaded mode is the default.
17.1 - Multithreading concepts
This section is a brief overview on how AxoSyslog works in multithreaded mode. It is mainly for illustration purposes: the concept has been somewhat simplified and may not completely match reality.
Note
The way AxoSyslog uses multithreading may change in future releases. The current documentation applies to version 4.5.
AxoSyslog always uses multiple threads:
A main thread that is always running
A number of worker threads that process the messages. You can influence the behavior of worker threads using the threaded() option and the --worker-threads command-line option.
Some other, special threads for internal functionalities. For example, certain destinations run in a separate thread, independently of the multithreading (threaded()) and --worker-threads settings of AxoSyslog.
The maximum number of worker threads AxoSyslog uses is the number of CPUs or cores in the host running AxoSyslog (up to 64). You can limit this value using the --worker-threads command-line option that sets the maximum total number of threads AxoSyslog can use, including the main AxoSyslog thread. However, the --worker-threads option does not affect the supervisor of AxoSyslog. The supervisor is a separate process (see The syslog-ng manual page), but certain operating systems might display it as a thread. In addition, certain destinations always run in a separate thread, independently of the multithreading (threaded()) and --worker-threads settings of AxoSyslog.
When an event requiring a new thread occurs (for example, AxoSyslog receives new messages, or a destination becomes available), AxoSyslog tries to start a new thread. If there are no free threads, the task waits until a thread finishes its task and becomes available. There are two types of worker threads:
Reader threads read messages from a source (as many as possible, but limited by the log-fetch-limit() and log-iw-size() options). The thread then processes these messages, that is, performs filtering, rewriting and other tasks as necessary, and puts the log message into the queue of the destination. If the destination does not have a queue (for example, usertty), the reader thread sends the message to the destination, without the interaction of a separate writer thread.
Writer threads take the messages from the queue of the destination and send them to the destination, that is, write the messages into a file, or send them to the syslog server over the network. The writer thread starts to process messages from the queue only if the destination is writable, and there are enough messages in the queue, as set in the flush-lines() option. Writer threads stop processing messages when the destination becomes unavailable, or there are no more messages in the queue.
Sources and destinations affected by multithreading
The following list describes which sources and destinations can use multiple threads. Changing the --worker-threads command-line option changes the number of threads available to these sources and destinations.
The tcp and syslog(tcp) sources can process independent connections in separate threads. The number of independent connections is limited by the max-connections() option of the source. Separate sources are processed by separate thread, for example, if you have two separate tcp sources defined that receive messages on different IP addresses or port, AxoSyslog will use separate threads for these sources even if they both have only a single active connection.
The udp, file, and pipe sources use a single thread for every source statement.
The tcp, syslog, and pipe destinations use a single thread for every destination.
The file destination uses a single thread for writing the destination file, but may use a separate thread for each destination file if the filename includes macros.
Sources and destinations not affected by multithreading
The following list describes sources and destinations that use a separate thread even if you disable multithreading in AxoSyslog, in addition to the limit set in the --worker-threads command-line option.
The logstore destination uses separate threads for writing the messages from the journal to the logstore files, and also for timestamping. These threads are independent from the setting of the --worker-threads command-line option.
Every sql destination uses its own thread. These threads are independent from the setting of the --worker-threads command-line option.
The java destinations use one thread, even if there are multiple Java-based destinations configured. This thread is independent from the setting of the --worker-threads command-line option.
17.2 - Configuring multithreading
Starting with version 3.6, AxoSyslog runs in multithreaded mode by default. You can enable multithreading in AxoSyslog using the following methods:
Globally using the threaded(yes) option.
Separately for selected sources or destinations using the flags("threaded") option.
Example: Enabling multithreading
To enable multithreading globally, use the threaded option:
options { threaded(yes);};
To enable multithreading only for a selected source or destination, use the flags("threaded") option:
Destinations that have a queue process that queue in a single thread. Multiple sources can send messages to the same queue, so the queue can scale to multiple CPUs. However, when the writer thread writes the queue contents to the destination, it will be single-threaded.
Message parsing, rewrite rules, filters, and other types of message processing is performed by the reader thread in a sequential manner. This means that such operations can scale only if reading messages from the source can be multithreaded. For example, if a tcp source can process messages from different connections (clients) in separate threads. If the source cannot use multiple threads to process the messages, the operations will not scale.
To improve the processing power of AxoSyslog and scale to more processors, use the following methods:
To improve scaling on the source side, use more sources, for example, more source files, or receive the messages from more parallel connections. For network sources, you can also configure a part of your clients to send the messages to a different port of your AxoSyslog server, and use separate source definitions for each port.
On the destination side, when writing the log messages to files, use macros in the filename to split the messages to separate files (for example, using the ${HOST} macro). Files with macros in their filenames are processed in separate writer threads.
On the destination side, when sending messages to an AxoSyslog server, you can use multiple connections to the server if you configure the AxoSyslog server to receive messages on multiple ports, and configure separate destinations on the clients to use both ports.
18 - Nonsequential message processing
By default, AxoSyslog processes log messages arriving from a single connection sequentially. Sequential processing:
ensures message ordering, and the
efficient use CPU on a per message basis.
Sequential processing performs well if you have relatively many parallel connections, in which case it uses all the available CPU cores. However, if a small number of connections deliver a large number of messages, this behavior becomes a bottleneck.
Starting with AxoSyslog version 4.3, AxoSyslog can split a stream of incoming messages into a set of partitions, which can be processed by multiple threads in parallel. Depending on how you partition the stream, you might lose the message ordering, but can scale the incoming load to all CPUs in the system, even if the entire load is coming from a single, chatty sender.
To enable this mode of execution, use the parallelize() element in your log path.
The following example takes the messages of the tcp() source and processes them with 4 parallel threads, regardless of the number of connections used to deliver the messages to the tcp() source.
log {source{ tcp( port(2000) log-iw-size(10M) max-connections(10) log-fetch-limit(100000));}; parallelize(partitions(4));# from this part on, messages are processed in parallel even if# messages are originally coming from a single connection parser { ... }; destination { ... };};
parallelize() uses round-robin to allocate messages to partitions by default, but you can retain ordering for a subset of messages with the partition-key() option. The partition-key() option specifies a template: messages that expand the template to the same value are mapped to the same partition. For example, you can partition messages based on their sender host:
log {source{ tcp( port(2000) log-iw-size(10M) max-connections(10) log-fetch-limit(100000));}; parallelize(partitions(4) partition-key("$HOST"));# from this part on, messages are processed in parallel if their# $HOST value differs. Messages with the same $HOST will be mapped# to the same partition and are processed sequentially. parser { ... }; destination { ... };};
19 - Writing Python modules
AxoSyslog has comprehensive support for implementing various logging components in Python. This chapter shows you how to use this functionality.
When to use Python
The Python bindings are useful if the facilities provided by the AxoSyslog configuration language is not sufficient, that is:
AxoSyslog doesn’t support a specific service (for example, API based log sources or information sources that you want to use for data enrichment purposes).
The AxoSyslog configuration language does not support a specific transformation (in that case, please tell us about your use-case).
You want to work on complex data structures (for example, JSON) which easier to do in a real programming language.
While Python is very powerful and you can produce clean and production ready solutions with it, the drawback is usually performance. Python code is usually slower than the native functionality that AxoSyslog provides.
To offset this impact of performance degradation, it’s a good strategy to only process a subset of the incoming log stream with Python code and use native configuration elements to select which subset is traversing said Python code.
Creating and storing the Python code
You can embed Python code directly into syslog-ng.conf, or work with Python modules.
Embedding Python into AxoSyslog configuration
You can simply use a top-level python {} block to embed your Python code, like this:
@version: 4.0
python {def template_function(msg):
return b"Hello World from Python! Original message: " + msg['MSGHDR'] + msg['MESSAGE']};log {source{ tcp(port(2000));}; destination { file("logfile" template("$ISODATE$(python template_function)"));};};
Using Python modules
You can also put your code into a proper Python module and then use it from there. AxoSyslog automatically adds ${sysconfdir}/python to your PYTHONPATH (normally /etc/syslog-ng/python), with that in mind add the following code to /etc/syslog-ng/python/mytemplate.py:
deftemplate_function(msg):returnb"Hello World from Python! Original message: "+msg['MSGHDR']+msg['MESSAGE']
The Python glue in AxoSyslog automatically imports modules when it encounters an identifier in dotted notation, so if you use this AxoSyslog config:
AxoSyslog recognizes that mytemplate.template_function is a qualified name and attempts to import mytemplate as a module and then looks up template_function within that module.
Note: Modules are only imported once, so you will need to restart AxoSyslog for a change to take effect.
AxoSyslog reload and Python
When you reload syslog-ng (with syslog-ng-ctl reload or systemctl reload syslog-ng) then the python block in your configuration is reloaded with the rest of the configuration file. Any changes you make in Python code directly embedded in your configuration takes effect after the reload. This also means that any global variables are reset, so you cannot store state across reloads in your python {} block.
Modules are only imported once and kept across reloads, even if the AxoSyslog configuration is reloaded. This means that you can store global state in modules and they will be kept, even as AxoSyslog reinitializes the configuration.
In case you want to reload a module every time AxoSyslog configuration is reinitialized, you need to do this explicitly with a code similar to this:
python{importmymoduleimportimportlib# reload mymodule every time syslog-ng reloadsimportlib.reload(mymodule)};
Destination driver
You can derive a destination driver in Python from the LogDestination class, as defined by the syslogng module, like in the following example:
The interface of the LogDestination class is documented in the syslogng.dest module, which is stored in the modules/python-modules/syslogng/dest.py file of the source tree.
Once all required methods are implemented, you can use the python destination in the AxoSyslog configuration language.
There’s a more complete example destination in the python_example() destination plugin, that is located in the directory modules/python-modules/syslogng/modules/example/ directory within the source tree, or the same files installed under ${exec_prefix}/syslog-ng/python/syslogng/modules in a production deployment.
Template function plugin
Template functions extend the AxoSyslog template language. They get a LogMessage object and return a string which gets embedded into the
output of the template. You can have AxoSyslog call a Python function from the template language using the $(python) template function.
@version: 4.0
python {def template_function(msg):
return b"Hello World from Python! Original message: " + msg['MSGHDR'] + msg['MESSAGE']};...
destination d_file { file("/var/log/whatever" template("$(python template_function)"));};
The Python function must be callable. IT receives a LogMessage instance and returns a string (str or bytes). The message passed to the template function is read-only. If you are trying to change a name-value pair, you will receive an exception.
Parser plugin
You can derive parser plugins in Python from the LogParser class as this example shows:
In contrast to template functions, parsers receive a read-writable LogMessage object, so you can modify its contents.
Source plugins
There are two kinds of source plugins that can be implemented in Python:
LogFetcher: LogFetcher provides a convenient interface for fetching messages from backend services via blocking APIs, but it is limited to performing the fetching operation in a sequential manner: you fetch a batch of messages, feed them to the AxoSyslog pipeline, then repeat.
LogSource: LogSource is more low-level but allows the use of an asynchronous framework (for example, asyncio) to perform message fetching along multiple threads of execution.
Both are defined by the syslogng.source module.
Source driver based on LogFetcher
LogFetcher provides a convenient interface for fetching messages from backend services via blocking APIs, but it is limited to performing the fetching operation in a sequential manner: you fetch a batch of messages, feed them to the AxoSyslog pipeline, then repeat.
For a LogFetcher class, you have to implement the fetch() method. This is the main entry point, which is automatically invoked by syslog-ng, whenever it consumes incoming messages.
This example generates one message every second, based on a literal string that is parsed as a syslog message.
The source is running in a dedicated thread, so it is free to block.
To limit the rate of generating messages, the time.sleep(1) call in the first line of the fetch() method sleeps for 1 second between the invocations of the method. If that sleep wasn’t there, the source would produce about 100-110k messages per second, depending on the speed of your CPU, the performance of the Python interpreter and the syslog-ng core.
If the fetcher connects to an external API, the sleep is usually not needed, as the response time of the API is a limiting factor.
Adding persistent state
If you are fetching messages from an API, you need to keep track of which messages were already fetched. Storing the position in a variable is not a good solution, because the value of the variable is lost when syslog-ng is reloaded or restarted (depending on where you store that variable, in the python {} block or in a module).
Use the Persist() class that uses the persistent state handling functionality of AxoSyslog. This allows you to persist variables in a file that gets stored in the ${localstatedir}/syslog-ng.persist file along with the rest of the syslog-ng states.
Once initialized, a Persist() instance behaves as a dict where you can store Python values. Currently str, bytes and int are supported. Anything you store in a persist instance is remembered even across restarts. The entries are backed up to disk immediately after changing them (using an mmap()-ped file), so you don’t have to explicitly commit them to disk.
You can store position information in a Persist() entry, but it’s not always the best choice. In AxoSyslog, producing messages is decoupled from their delivery: sometimes a message is still in-flight for a while before being delivered. This time can be significant if a destination consumes messages at a slow rate. In this case, if you store the position once fetched, the message would still be sitting in a queue waiting to be delivered. If the queue is not backed by a disk-buffer, then these messages would be lost, once syslog-ng is restarted.
The bookmarking mechanism allows messages to carry individual markers that uniquely identify a message and its position in a source stream. For example, in a source file the bookmark would contain the position of the message within that file. An API may have a similar mechanism in place in which the source API associates an opaque to each message, which signifies its position in the repository.
A specific example for bookmarks is systemd-journald, which has a “cursor” indicating the position of each journal record. The cursor can be used to restart the reading of the log stream.
Once you’ve identified what mechanism the source offers that maps to the bookmark concept, decide how you want to track these bookmarks. Which bookmark tracking strategy you should use depends on the API specifics.
Some APIs are sequential in nature, thus you can only acknowledge the “last fetch position” in that sequence.
Other APIs allow you to acknowledge messages individually.
AxoSyslog supports both methods.
The following Python example updates the current position in a source stream only when the AxoSyslog destination has acknowledged the messages in the sequence (that is, when the messages were properly sent).
classMyFetcher(LogFetcher):counter=0definit(self,options):self.persist=Persist("MyFetcher_persistent_data",defaults={"position":0})self.counter=self.persist['position']# pass self.message_acked method as ACK callbackself.ack_tracker=ConsecutiveAckTracker(ack_callback=self.message_acked)returnTruedefmessage_acked(self,acked_message_bookmark):# update current persisted position when syslog-ng delivered the# message, but only then.self.persist['position']=acked_message_bookmarkdeffetch(self):time.sleep(1)self.counter+=1# depending on the speed of our consumer and the setting of# flags(flow-control), the current counter and the acked value may# differ in the messages generated.msg=LogMessage.parse("<5>2022-02-02T10:23:45+02:00 HOST program[pid]: foobar %d (acked so far %d)"%(self.counter,self.persist['position']),self.parse_options)# this is where we set the bookmark for the messagemsg.set_bookmark(self.counter)returnself.SUCCESS,msg
Acknowledgement tracking strategies
Some APIs provide simple, while others provide more complex ways to track messages that are processed. AxoSyslog provides the following strategies to cope with them.
Instant tracking (InstantAckTracker): Messages are considered delivered as soon as the destination driver (or the disk-buffer) acknowledges them. Out-of-order deliveries are reported as they happen, so an earlier message may be acknowledged later than a message originally encountered later in the source stream.
Consecutive tracking (ConsecutiveAckTracker): Messages are assumed to form a stream and the bookmark is a position in that stream. Unordered deliveries are properly handled by only acknowledging messages that were delivered in order. If unordered delivery happens, the tracker waits for the sequence to fill up, that is, it waits for all preceeding messages to be delivered as well.
Batched tracking (BatchedAckTracker): Messages are assumed to be independent, not forming a sequence of events. Each message is individually tracked, the source driver has the means to get delivery notifications of each and every message independently. The acknowledgements are accumulated until a timeout happens, at which point they get reported as a single batch.
You can initialize your ack_tracker in the init method, like this:
classMyFetcher(LogFetcher):...definit(self,options):# pass self.message_acked method as ACK callbackself.ack_tracker=ConsecutiveAckTracker(ack_callback=self.message_acked)returnTruedefmessage_acked(self,acked_message_bookmark):passdeffetch(self):...msg.set_bookmark("whatever-bookmark-value-that-denotes-position")...
The previous example uses ConsecutiveAckTracker, so you get acknowledgements in the order messages were generated. The argument of the message_acked callback is the “bookmark” value that you set using set_bookmark().
Using InstantAckTracker is very similar, just replace ConsecutiveAckTracker with InstantAckTracker. In this case you’d get a callback as soon as a message is delivered without preserving the original ordering.
While ConsecutiveAckTracker() seems to provide a much more useful service, InstantAckTracker() performs better, as it does not have to track acknowledgements of individual messages.
The most complex scenario is implemented by BatchedAckTracker, this allows you to track the acknowledgements for individual messages, as they happen, not enforcing any kind of ordering.
BatchedAckTracker calls your callback periodically, as set by the timeout argument in milliseconds. batch_size specifies the number of outstanding messages at a time.
With this interface it’s quite easy to send acknowledgements back to the source interface where per-message acknowledgements are needed (for example, Google PubSub).
Accessing the flags() option
The state of the flags() option is mapped to the self.flags variable, which is a Dict[str, bool], for example:
LogSource allows the use of an asynchronous framework (for example, asyncio) to perform message fetching along multiple threads of execution.
The following example uses asyncio to generate two independent sequences of messages: the first is generated every second, the other every 1.5 seconds, running concurrently via an asyncio event loop.
It is also easy to create a source that implements an HTTP server, and which injects messages coming via HTTP to the AxoSyslog pipeline.
While this works, the syntax doesn’t look like other parts of the configuration, and is also hard to read. The usual syntax for referencing regular drivers is something like this:
To make the syntax more native, you can use the block functionality to wrap your Python driver, hide that it’s actually Python, and provide a syntax to your code that is more convenient to use.
This block allows the use of the more AxoSyslog-native syntax, completely hiding the fact that the implementation is Python based, concentrating on functionality.
Add this wrapper to your Python module in an scl subdirectory as a file with a .conf extension. AxoSyslog automatically includes these files along the rest of the SCL.
Adding the code to syslog-ng
To add your Python-based modules to syslog-ng, complete the following steps.
Create a Python package: add the __init__.py file and anything that the file references to the modules/python-modules/<name-of-your-module> directory of the syslog-ng repository.
Add your files to the source tarball by listing them in the EXTRA_DIST variable of the modules/python-modules/Makefile.am file.
Run make install to install your module along the rest of the syslog-ng binaries.
Open a pull request.
External dependencies
If your Python code depends on third-party libraries, those need to be installed on the system where your code is deployed. If your deployment mechanism is based on DEB or RPM packages, make sure that you add these OS-level dependencies to the packages generated.
For DEB packages, add the dependency package to the Depends line.
For RPM packages, add the dependency package as a Requires line to the .spec file.
If you want to use pip/requirements.txt to deploy dependencies, you can invoke pip during make install time so that AxoSyslog’s private Python directory would contain all the dependencies that you require.
Adding Python code to the DEB package
To add your module to the syslog-ng DEB package, complete the following steps.
Create a new file in packaging/debian/ called syslog-ng-mod-<yourmodule>.install.
Populate this file with wildcard patterns that capture the files of your package after installation. For example:
Package:syslog-ng-mod-<yourmodule>Architecture:anyMulti-Arch:foreignDepends:${shlibs:Depends}, ${misc:Depends}, syslog-ng-core (>= ${source:Version}), syslog-ng-core (<< ${source:Version}.1~), syslog-ng-mod-pythonDescription:The short description of the packageThis is a longer description with dots separating paragraphs..This package provides a collection of example plugins.
Add your .install file to the tarball by adding it to the EXTRA_DIST in the Makefile.am.
Adding Python code to RPM packages
The RPM package is less modular than the Debian package and it automatically captures all Python modules in the syslog-ng-python package without having to list them explicitly.
If you need to customize the installation, you can find the spec file in packaging/rhel/syslog-ng.spec which is populated and copied to the root at tarball creation.
20 - Best practices and examples
This chapter discusses some special examples and recommendations.
20.1 - General recommendations
This section provides general tips and recommendations on using syslog-ng. Some of the recommendations are detailed in the sections below:
Do not base the separation of log messages in different files on the facility parameter. As several applications and processes can use the same facility, the facility does not identify the application that sent the message. By default, the facility parameter is not even included in the log message itself. In general, sorting the log messages into several different files can make finding specific log messages difficult. If you must create separate log files, use the application name.
Standard log messages include the local time of the sending host, without any time zone information. It is recommended to replace this timestamp with an ISODATE timestamp, because the ISODATE format includes the year and timezone as well. To convert all timestamps to the ISODATE format, include the following line in the syslog-ng.conf configuration file:
options {ts-format(iso);};
Resolving the IP addresses of the clients to domain names can decrease the performance. For details, see Using name resolution in syslog-ng.
20.2 - Handling large message load
This section provides tips on optimizing the performance of syslog-ng. Optimizing the performance is important for AxoSyslog hosts that handle large traffic.
Do not use the usertty() destination driver. Under heavy load, the users are not be able to read the messages from the console, and it slows down syslog-ng.
Do not use regular expressions in our filters. Evaluating general regular expressions puts a high load on the CPU. Use simple filter functions and logical operators instead. For details, see Regular expressions.
Warning
When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the AxoSyslog server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf() option of the source is increased. In this cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.
As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152 bytes.
Increase the value of the flush-lines() parameter. Increasing flush-lines() from 0 to 100 can increase the performance of AxoSyslog by 100%.
20.3 - Using name resolution in syslog-ng
The AxoSyslog application can resolve the hostnames of the clients and include them in the log messages. However, the performance of AxoSyslog is severely degraded if the domain name server is unaccessible or slow. Therefore, it is not recommended to resolve hostnames in syslog-ng. If you must use name resolution from syslog-ng, consider the following:
Use DNS caching. Verify that the DNS cache is large enough to store all important hostnames. (By default, the AxoSyslog DNS cache stores 1007 entries.)
options { dns-cache-size(2000);};
If the IP addresses of the clients change only rarely, set the expiry of the DNS cache large.
Domain name resolution is important mainly in relay and server mode.
20.3.1 - Resolving hostnames locally
Purpose:
Resolving hostnames locally enables you to display hostnames in the log files for frequently used hosts, without having to rely on a DNS server. The known IP address - hostname pairs are stored locally in a file. In the log messages, AxoSyslog will replace the IP addresses of known hosts with their hostnames. To configure local name resolution, complete the following steps:
Steps:
Add the hostnames and the respective IP addresses to the file used for local name resolution. On Linux and UNIX systems, this is the /etc/hosts file. Consult the documentation of your operating system for details.
Instruct AxoSyslog to resolve hostnames locally. Set the use-dns() option to persist_only.
Set the dns-cache-hosts() option to point to the file storing the hostnames.
To collect logs from a chroot using an AxoSyslog client running on the host, complete the following steps:
Steps:
Create a /dev directory within the chroot. The applications running in the chroot send their log messages here.
Create a local source in the configuration file of the AxoSyslog application running outside the chroot. This source should point to the /dev/log file within the chroot (for example, to the /chroot/dev/log directory).
Include the source in a log statement.
Note
You need to set up timezone information within your chroot as well. This usually means creating a symlink to /etc/localtime.
20.5 - Configuring log rotation
The AxoSyslog application does not rotate logs by itself. To use AxoSyslog for log rotation, consider the following approaches:
Use logrotate together with AxoSyslog:
It is ideal for workstations or when processing fewer logs.
It is included in most distributions by default.
Less scripting is required, only logrotate has to be configured correctly.
Requires frequent restart (AxoSyslog must be reloaded/restarted when the files are rotated). After rotating the log files, reload AxoSyslog using the syslog-ng-ctl reload command, or use another method to send a SIGHUP to AxoSyslog.
The statistics collected by AxoSyslog, and the correlation information gathered with Pattern Database, are lost with each restart.
Separate incoming logs based on time, host or other information:
It is ideal for central log servers, where regular restart of AxoSyslog is unfavorable.
Requires shell scripts or cron jobs to remove old logs.
It can be done by using macros in the destination name (in the filename, directory name, or the database table name). (For details on using macros, see Templates and macros.)
Example: File destination for log rotation
This sample file destination configuration stores incoming logs in files that are named based on the current year, month and day, and places these files in directories that are named based on the hostname:
This sample logstore destination configuration stores incoming logs in logstores that are named based on the current year, month and day, and places these logstores in directories that are named based on the hostname:
This sample command for cron removes files older than two weeks from the /var/log/remote directory:
find /var/log/remote/ -daystart -mtime +14 -type f -exec rm {}\;
20.6 - Load balancing logs between multiple destinations
These sections describe a method of load balancing logs between multiple AxoSyslog destinations. The first subsection describes the round robin load balancing method based on the R_MSEC macro of AxoSyslog, while the second subsection describes a configuration generator that you can use as an alternative to using the example configuration described in the first subsection.
For more information about the R_MSEC macro and further macros of AxoSyslog, see Macros of AxoSyslog.
20.6.1 - Load balancing with a round robin load balancing method based on the R_MSEC macro of syslog-ng OSE
This section describes a round robin load balancing method based on the R_MSEC macro of AxoSyslog to load balance your logs between multiple AxoSyslog destinations.
Note
If R_MSEC is not precise enough, you can replace it with R_USEC (which uses microseconds instead of milliseconds).
For more information about the R_MSEC macro and further macros of AxoSyslog, see Macros of AxoSyslog.
Example: round robin load balancing between multiple destinations
The following example is a round-robin load balancing method, based on AxoSyslog’s R_MSEC macro.
The filter {" <return value >" == "$(% ${R_MSEC} 2)"}; code snippets (in bold) serve as the basis of the method. This filter separates incoming log messages’ timestamp values based on the R_MSEC macro, using a division with remainder method, and distributes the log messages equally between two destinations based on the return value (in this case, 0 or 1).
If you need a file instead of a network destination, replace the network destination with the file in the example (and use the same analogy for any other AxoSyslog destinations).
20.6.2 - Configuration generator for the load balancing method based on MSEC hashing
This section describes a configuration generator for the load balancing method based on MSEC hashing to load balance your logs between multiple AxoSyslog destinations.
Also consider that the configuration generator script may change incompatibly in the future. As a result, Axoflow does not officially support using this script, and recommends that you only use this script at your own risk.
Where destinations share the same configuration except for the destination address, balancing is based on MSEC hashing.
21 - Troubleshooting
This chapter provides tips and guidelines about troubleshooting problems related to syslog-ng.
As a general rule, first try to log the messages to a local file. Once this is working, you know that AxoSyslog is running correctly and receiving messages, and you can proceed to forwarding the messages to the server.
Always check the configuration files for any syntax errors on both the client and the server using the syslog-ng --syntax-only command.
If the AxoSyslog server does not receive the messages, verify that the IP addresses and ports are correct in your sources and destinations. Also, check that the client and the server uses the same protocol (a common error is to send logs on UDP, but configure the server to receive logs on TCP).
If the problem persists, use tcpdump or a similar packet sniffer tool on the client to verify that the messages are sent correctly, and on the server to verify that it receives the messages.
To find message-routing problems, run AxoSyslog with the following command syslog-ng -Fevd. That way AxoSyslog will run in the foreground, and display debug messages about the messages that are processed.
If AxoSyslog is closing the connections for no apparent reason, be sure to check the log messages of syslog-ng. You may also want to run syslog-ng with the --verbose or --debug command-line options for more-detailed log messages. You can enable these messages without restarting syslog-ng using the syslog-ng-ctl verbose --set=on command. For details, see the The syslog-ng.conf manual page.
Build up encrypted connections step-by-step. First create a working, unencrypted (for example, TCP) connection, then add TLS encryption, and finally, client authentication if needed.
If you use the same driver and options in the destination of your AxoSyslog client and the source of your AxoSyslog server, everything should work as expected. Unfortunately, there are some other combinations, that may seem to work, but result in losing parts of the messages. For details on the working combinations, see Things to consider when forwarding messages between AxoSyslog hosts.
Support
In case you need help with any of the AxoSyslog projects, or directly with syslog-ng, you have several ways to contact us:
We also provide consulting and professional services for logging and observability related projects. Contact us if you need our help!
21.1 - Possible causes of losing log messages
During the course of a message from the sending application to the final destination of the message, there are a number of locations where a message may be lost, even though AxoSyslog does its best to avoid message loss. Usually losing messages can be avoided with careful planning and proper configuration of AxoSyslog and the hosts running syslog-ng. The following list shows the possible locations where messages may be lost, and provides methods to minimize the risk of losing messages:
Between the application and the AxoSyslog client: Make sure to use an appropriate source to receive the logs from the application (for example, from /dev/log). For example, use unix-stream instead of unix-dgram whenever possible.
When AxoSyslog is sending messages: If AxoSyslog cannot send messages to the destination and the output buffer gets full, AxoSyslog will drop messages.
Use flags (flow-control) to avoid this (for details, see Configuring flow-control). For more information about the error caused by the missing flow-control, see Destination queue full in Error messages.
The number of dropped messages is displayed per destination in the log message statistics of AxoSyslog (for details, see Statistics of AxoSyslog).
On the network: When transferring messages using the UDP protocol, messages may be lost without any notice or feedback — such is the nature of the UDP protocol. Always use the TCP protocol to transfer messages over the network whenever possible.
For details on minimizing message loss when using UDP, see the tutorial.
In the socket receive buffer: When transferring messages using the UDP protocol, the UDP datagram (that is, the message) that reaches the receiving host placed in a memory area called the socket receive buffer. If the host receives more messages than it can process, this area overflows, and the kernel drops messages without letting AxoSyslog know about it. Using TCP instead of UDP prevents this issue. If you must use the UDP protocol, increase the size of the receive buffer using the so-rcvbuf() option.
When AxoSyslog is receiving messages:
The receiving AxoSyslog (for example, the AxoSyslog server or relay) may drop messages if the fifo of the destination file gets full. The number of dropped messages is displayed per destination in the log message statistics of AxoSyslog (for details, see Statistics of AxoSyslog).
When the destination cannot handle large load: When AxoSyslog is sending messages at a high rate into an SQL database, a file, or another destination, it is possible that the destination cannot handle the load, and processes the messages slowly. As a result, the buffers of AxoSyslog fill up, AxoSyslog cannot process the incoming messages, and starts to loose messages. For details, see the previous entry. Use the throttle parameter to avoid this problem.
As a result of an unclean shutdown of the AxoSyslog server: If the host running the AxoSyslog server experiences an unclean shutdown, it takes time until the clients realize that the connection to the AxoSyslog server is down. Messages that are put into the output TCP buffer of the clients during this period are not sent to the server.
When AxoSyslog is writing messages into files: If AxoSyslog receives a signal (SIG) while writing log messages to file, the log message that is processed by the write call can be lost if the flush_lines parameter is higher than 1.
21.2 - Creating core files
Purpose:
When syslog-ng crashes for some reason, it can create a core file that contains important troubleshooting information. To enable core files, complete the following procedure:
Steps:
Core files are produced only if the maximum core file size ulimit is set to a high value in the init script of syslog-ng. Add the following line to the init script of syslog-ng:
ulimit -c unlimited
Verify that syslog-ng has permissions to write the directory it is started from, for example, /opt/syslog-ng/sbin/.
If syslog-ng crashes, it will create a core file in the directory syslog-ng was started from.
To test that syslog-ng can create a core file, you can create a crash manually. For this, determine the PID of syslog-ng (for example, using the ps -All|grep syslog-ng command), then issue the following command: kill -ABRT <syslog-ng pid>
This should create a core file in the current working directory.
21.3 - Collecting debugging information with strace, truss, or tusc
To properly troubleshoot certain situations, it can be useful to trace which system calls AxoSyslog performs. How this is performed depends on the platform running AxoSyslog. In general, note the following points:
When AxoSyslog is started, a supervisor process might stay in the foreground, while the actual syslog-ng daemon goes to the background. Always trace the background process.
Apart from the system calls, the time between two system calls can be important as well. Make sure that your tracing tool records the time information as well. For details on how to do that, refer to the manual page of your specific tool (for example, strace on Linux, or truss on Solaris and BSD).
Run your tracing tool in verbose mode, and if possible, set it to print long output strings, so the messages are not truncated.
When using strace, also record the output of lsof to see which files are accessed.
The following are examples for tracing system calls of syslog-ng on some platforms. The output is saved into the /tmp/syslog-ng-trace.txt file, sufficed with the PID of the related syslog-ng process. The path of the syslog-ng binary may be different for your installation, as distribution-specific packages may use different paths.
IBM AIX and Solaris: truss -f -o /tmp/syslog-ng-trace.txt -r all -w all -u libc:: /opt/syslog-ng/sbin/syslog-ng -d -d -d
Note
To execute these commands on an already running AxoSyslog process, use the -p <pid_of_syslog-ng> parameter.
21.4 - Running a failure script
Purpose:
You can create a failure script that is executed when AxoSyslog terminates abnormally, that is, when it exits with a non-zero exit code. For example, you can use this script to send an automatic email notification.
Prerequisites:
The failure script must be the following file: /opt/syslog-ng/sbin/syslog-ng-failure, and must be executable.
To create a sample failure script, complete the following steps.
Steps:
Create a file named /opt/syslog-ng/sbin/syslog-ng-failure with the following content:
Make the file executable: chmod +x /opt/syslog-ng/sbin/syslog-ng-failure
Run the following command in the /opt/syslog-ng/sbin directory: ./syslog-ng --process-mode=safe-background; sleep 0.5; ps aux | grep './syslog-ng' | grep -v grep | awk '{print $2}' | xargs kill -KILL; sleep 0.5; cat /tmp/test.txt
The command starts AxoSyslog in safe-background mode (which is needed to use the failure script) and then kills it. You should see that the relevant information is written into the /tmp/test.txt file, for example:
Thu May 18 12:08:58 UTC 2017 Name............syslog-ng
Chroot dir......NULL
Pid file dir....NULL
Pid file........NULL
Cwd.............NULL
Caps............NULL
Reason..........signalled
Argbuf..........9
Restarting......not-restarting
You should also see messages similar to the following in system syslog. The exact message depends on the signal (or the reason why AxoSyslog stopped):
May 18 13:56:09 myhost supervise/syslog-ng[10820]: Daemon exited gracefully, not restarting;exitcode='0' May 18 13:57:01 myhost supervise/syslog-ng[10996]: Daemon exited due to a deadlock/signal/failure, restarting;exitcode='131' May 18 13:57:37 myhost supervise/syslog-ng[11480]: Daemon was killed, not restarting;exitcode='9'
The failure script should run on every non-zero exit event.
21.5 - Stopping the syslog-ng process
To avoid problems, always use the init scripts to stop syslog-ng (/etc/init.d/syslog-ng stop), instead of using the kill command. This is especially true on Solaris and HP-UX systems, here use /etc/init.d/syslog stop.
21.6 - Reporting bugs and finding help
If you need help, want to open a support ticket, or report a bug, we recommend using the syslog-ng-debun tool to collect information about your environment and AxoSyslog version. For details, see the The syslog-debun manual page. For support contacts, see Getting support.
21.7 - Recover data from orphaned diskbuffer files
When you change the configuration of a AxoSyslog host that uses disk-based buffering (also called disk queue), AxoSyslog may start new disk buffer files for the destinations that you have changed. In this case, AxoSyslog abandons the old disk queue files. If there were unsent log messages in the disk queue files, these messages remain in the disk queue files, and will not be sent to the destinations.
21.8 - No local logs after specifying an unusual storage directory
Security-Enhanced Linux (SELinux) is a set of kernel and user-space tools enforcing strict access control policies. SELinux rules in Linux distributions cover all aspects of the configuration coming in the syslog-ng package available in the distribution. But as soon as an unusual port number or directory name is specified in the configuration, syslog-ng fails to work even with a completely legitimate configuration.
When you choose to save logs of a central AxoSyslog server to a directory other than the /var/log directory, logs will not start appearing on the newly configured directory. For details on how to fix this issue, see section “Using a different storage directory” in the blog post titled Using syslog-ng with SELinux in enforcing mode.
21.9 - No logs after specifying an unusual port number
Security-Enhanced Linux (SELinux) is a set of kernel and user-space tools enforcing strict access control policies. SELinux rules in Linux distributions cover all aspects of the configuration coming in the syslog-ng package available in the distribution. But as soon as an unusual port number or directory name is specified in the configuration, syslog-ng fails to work even with a completely legitimate configuration.
By default, SELinux only allows connections to the default syslog ports. When you have to use any other port for some reason, sending logs to that port will not work. For details on how to fix this issue, see section “Using a different port” in the blog post titled Using syslog-ng with SELinux in enforcing mode.
21.10 - Error messages
This section describes the most common error messages.
Flow-control must be enabled in the log path. When flow-control is enabled, syslog-ng will stop reading messages from the sources of the log statement if the destinations are not able to process the messages at the required speed.
If flow-control is enabled, syslog-ng will only drop messages if the destination queues/window sizes are improperly sized.
Solution:
Enable flow-control in the log path.
If flow-control is disabled, syslog-ng will drop messages if the destination queues are full. Note that syslog-ng will drop messages even if the server is alive. If the remote server accepts logs at a slower rate than the sender syslog-ng receives them, the sender syslog-ng will fill up the destination queue, then drop the newer messages. Sometimes this error occurs only at a specific time interval, for example, only between7:00AM and8:00AM or between16:00PM and17:00PM when your users log in or log off and that generates a lot of messages within a short interval.
The error message is displayed when using Transport Layer Security (TLS). The syslog-ng application uses OpenSSL for TLS and this message indicates that the certificate contains characters that OpenSSL cannot process.
The error occurs when the certificate comes from Windows and you want to use it on a Linux-based computer. On Windows, the end of line (EOL) character is different (\r\n) compared to Linux (\n).
To verify this, open the certificate in a text editor, for example, MCEdit. Notice the ^M characters as shown in the image below:
Solution:
On Windows, save the certificate using UTF-8, for example, using Notepad++.
Windows Notepad is not able to save the file in normal UTF-8, even if you select it.
In Notepad++, from the menu, selectEncoding.
Change the value fromUTF-8-BOMtoUTF-8.
Save.
On Linux, run dos2unix cert.pem. This will convert the file to a Linux-compatible style.
Alternatively, replace the EOL characters in the file manually.
21.11 - SELinux prevents using the execmem access on a process
If you are using a recent enough PCRE library, AxoSyslog will automatically use the JIT of the regexp engine, which will result in a similar error:
setroubleshoot [21631] : SELinux is preventing <syslog-ng path> from using the execmem access on a process. (...) python [21631] : SELinux is preventing <syslog-ng path> from using the execmem access on a process.
To resolve this issue, switch off the PCRE JIT compile function by using the disable-jitflags() option in the given filter or rewrite rule of your configuration.
22 - Getting support
In case you need help with any of the AxoSyslog projects, or directly with syslog-ng, you have several ways to contact us:
We also provide consulting and professional services for logging and observability related projects. Contact us if you need our help!
23 - Manual pages
23.1 - The `syslog-ng.conf` manual page
Name
syslog-ng.conf — configuration file
Synopsis
syslog-ng.conf
Description
The AxoSyslog application is a flexible and highly scalable system logging application. Typically, AxoSyslog is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices - called syslog-ng clients - all run AxoSyslog, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote AxoSyslog server, where the server sorts and stores them.
Basic concepts of AxoSyslog
The AxoSyslog application reads incoming messages and forwards them to the selected destinations. The AxoSyslog application can receive messages from files, remote hosts, and other sources.
Log messages enter AxoSyslog in one of the defined sources, and are sent to one or more destinations.
Sources and destinations are independent objects, log paths define what AxoSyslog does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations: messages arriving from a source are sent to every destination listed in the log path. A log path defined in AxoSyslog is called a log statement.
Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, AxoSyslog sends only the messages satisfying the filter rules to the destinations set in the log path.
Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.
Configuring AxoSyslog
The main body of the configuration file consists of object definitions: sources, destinations, logpaths define which log message are received and where they are sent. All identifiers, option names and attributes, and any other strings used in the syslog-ng.conf configuration file are case sensitive. Object definitions (also called statements) have the following syntax:
Type of the object: One of source, destination, log, filter, parser, rewrite rule, or template.
Identifier of the object: A unique name identifying the object. When using a reserved word as an identifier, enclose the identifier in quotation marks ("").
All identifiers, attributes, and any other strings used in the AxoSyslog configuration file are case sensitive.
Note
Use identifiers that refer to the type of the object they identify. For example, prefix source objects with s_, destinations with d_, and so on.
Repeating a definition of an object (that is, defining the same object with the same id more than once) is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file.
Parameters: The parameters of the object, enclosed in braces {parameters}.
Semicolon: Object definitions end with a semicolon (;).
For example, the following line defines a source and calls it s_internal.
source s_internal { internal();};
The object can be later referenced in other statements using its ID, for example, the previous source is used as a parameter of the following log statement:
log { source(s_internal); destination(d_file);};
The parameters and options within a statement are similar to function calls of the C programming language: the name of the option followed by a list of its parameters enclosed within brackets and terminated with a semicolon.
For example, the file() driver in the following source statement has three options: the filename (/var/log/apache/access.log), follow-freq(), and flags(). The follow-freq() option also has a parameter, while the flags() option has two parameters.
Objects may have required and optional parameters. Required parameters are positional, meaning that they must be specified in a defined order. Optional parameters can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object.
Example: Using required and optional parameters
The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect:
The AxoSyslog application has a number of global options governing DNS usage, the timestamp format used, and other general points. Each option may have parameters, similarly to driver specifications. To set global options add an option statement to the AxoSyslog configuration file using the following syntax:
The AxoSyslog application is a flexible and highly scalable system logging application. Typically, AxoSyslog is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices - called clients - all run AxoSyslog, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote AxoSyslog server, where the server sorts and stores them.
Options
--caps
Run AxoSyslog process with the specified POSIX capability flags.
If the --no-caps option is not set, AxoSyslog has been compiled with the --enable-linux-caps compile option, and the host supports CAP_SYSLOG, AxoSyslog uses the following capabilities: cap_net_bind_service, cap_net_broadcast, cap_net_raw, cap_dac_read_search, cap_dac_override, cap_chown, cap_fowner=p cap_syslog=ep
If the --no-caps option is not set, and the host does not support CAP_SYSLOG, AxoSyslog uses the following capabilities: cap_net_bind_service, cap_net_broadcast, cap_net_raw, cap_dac_read_search, cap_dac_override, cap_chown, cap_fowner=p cap_sys_admin=ep
Note that the capabilities are not case sensitive, the following command is also good: /opt/syslog-ng/sbin/syslog-ng -Fv --caps CAP_SYS_ADMIN,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_NET_BIND_SERVICE,CAP_FOWNER=pi
For details on the capability flags, see the following man pages: cap_from_text(3) and capabilities(7)
--cfgfile <file> or -f <file>
Use the specified configuration file.
--chroot <dir> or -C <dir>
Change root to the specified directory. The configuration file is read after chrooting so, the configuration file must be available within the chroot. That way it is also possible to reload the syslog-ng configuration after chrooting. However, note that the --user and --groupoptions are resolved before chrooting.
--check-startup: Use this option to perform a complete configuration initialization with syslog-ng, then exit with exit code indicating the result. You can use this option to check if the configuration is semantically valid and that syslog-ng can actually start. (The --syntax-only option catches only syntactical errors.)
You can also use syslog-ng --check-startup in a Kubernetes environment, to run it as a dedicated configuration check container.
Available only in AxoSyslog version 4.5 and later.
Using this option initializes things like network listeners, so it will probably not work if another syslog-ng instance running in the background (because in that case the network address is already in use).
--control <file> or -c <file>
Set the location of the syslog-ng control socket. Default value: /var/run/syslog-ng.ctl
--debug or -d
Start syslog-ng in debug mode.
--default-modules
A comma-separated list of the modules that are loaded automatically. Modules not loaded automatically can be loaded by including the @module <modulename> statement in the AxoSyslog configuration file. Available only in AxoSyslog version 4.1 and later.
--enable-core
Enable AxoSyslog to write core files in case of a crash to help support and debugging.
--fd-limit <number>
Set the minimal number of required file descriptors (fd-s). This sets how many files syslog-ng can keep open simultaneously. Default value: 4096. Note that this does not override the global ulimit setting of the host.
--foreground or -F
Do not daemonize, run in the foreground. When running in the foreground, AxoSyslog starts from the current directory ($CWD) so it can create core files (normally, AxoSyslog starts from /$PREFIX/var).
--group <group> or -g <group>
Switch to the specified group after initializing the configuration file.
--help or -h
Display a brief help message.
--log-level <level>
Set the internal log level of AxoSyslog to default, verbose, debug, or trace. Available in AxoSyslog 4.0 and later.
--module-registry
Display the list and description of the available modules. Note that not all of these modules are loaded automatically, only the ones specified in the --default-modules option.
--no-caps
Run AxoSyslog as root, without capability-support. This is the default behavior. On Linux, it is possible to run AxoSyslog as non-root with capability-support if AxoSyslog was compiled with the --enable-linux-caps option enabled. (Run syslog-ng --version to display the list of enabled build parameters.)
To run AxoSyslog with specific capabilities, use the --caps option.
--persist-file <persist-file> or -R <persist-file>
Set the path and name of the syslog-ng.persist file where the persistent options and data are stored.
--pidfile <pidfile> or -p <pidfile>
Set path to the PID file where the pid of the main process is stored.
--preprocess-into <output-file>
After processing the configuration file and resolving included files and variables, write the resulting configuration into the specified output file. Available only in AxoSyslog 4 F1 and later.
--process-mode <mode>
Sets how to run AxoSyslog: in the foreground (mainly used for debugging), in the background as a daemon, or in safe-background mode. By default, syslog-ng runs in safe-background mode. This mode creates a supervisor process called supervising syslog-ng, that restarts AxoSyslog if it crashes.
--stderr or -e
Log internal messages of AxoSyslog to stderr. Mainly used for debugging purposes in conjunction with the --foreground option. If not specified, syslog-ng logs such messages to its internal source.
--syntax-only or -s
Verify that the configuration file is syntactically correct and exit.
--user <user> or -u <user>
Switch to the specified user after initializing the configuration file (and optionally chrooting). Note that it is not possible to reload the syslog-ng configuration if the specified user has no privilege to create the /dev/log file.
--verbose or -v
Enable verbose logging used to troubleshoot AxoSyslog.
--version or -V
Display version number and compilation information, and also the list and short description of the available modules. For detailed description of the available modules, see the --module-registry option. Note that not all of these modules are loaded automatically, only the ones specified in the --default-modulesoption.
--worker-threads
Sets the number of worker threads AxoSyslog can use, including the main AxoSyslog thread. Note that certain operations in AxoSyslog can use threads that are not limited by this option. This setting has effect only when AxoSyslog is running in multithreaded mode. Available only in AxoSyslog 4 F1 and later. See The AxoSyslog 7 Administrator Guide for details.
dqtool — Display the contents of a disk-buffer file created with AxoSyslog.
Synopsis
dqtool [command] [options]
Description
Note
The dqtool application is distributed with the AxoSyslog system logging application, and is usually part of the AxoSyslog package.
The dqtool application is a utility that can be used to display and format the messages stored in a disk-buffer file.
The cat command
cat [options] [file]
Use the cat command to display the log messages stored in the disk-buffer (also called disk-queue) file, and also information from the header of the disk queue file. The messages are printed to the standard output (stdout), so it is possible to use grep and other tools to find particular log messages, for example, dqtool cat /var/log/messages.lgs |grep 192.168.1.1.
The cat command has the following options:
--debug or -d
Print diagnostic and debugging messages to stderr.
Use the relocate command to move or rename disk-buffer (also called disk-queue) files. Note that this option modifies the persist file. Stop AxoSyslog before using this command.
The cat command has the following options:
--all or -a
Relocate every disk-buffer file that is listed in the AxoSyslog persist file.
--new_path or -n
The directory where you want to move the disk-bufffer files. For example: /var/disk-buffers
--persist or -p
The path to the AxoSyslog persist file. The relocate command automatically updates the entries of the disk-buffer files in the persist file.
loggen — Generate syslog messages at a specified rate
Synopsis
loggen [options]
target [port]
Description
Note
The loggen application is distributed with the AxoSyslog system logging application, and is usually part of the AxoSyslog package.
The loggen application is a tool to test and stress-test your syslog server and the connection to the server. It can send syslog messages to the server at a specified rate using a number of connection types and protocols, including TCP, UDP, and unix domain sockets. The messages can be generated automatically (repeating the PADDstring over and over), or read from a file or the standard input.
When loggen finishes sending the messages, it displays the following statistics:
average rate: The average rate of the sent messages in messages/second.
count: The total number of messages sent.
time: The time required to send the messages in seconds.
average message size: The average size of the sent messages in bytes.
bandwidth: The average bandwidth used for sending the messages in kilobytes/second.
Options
--active-connections <number-of-connections>
Number of connections loggen will use to send messages to the destination. This option is usable only when using TCP or TLS connections to the destination. Default value: 1
The loggen utility waits until every connection is established before starting to send messages. See also the --idle-connections option.
--csv or -C
Send the statistics of the sent messages to stdout as CSV. This can be used for plotting the message rate.
--dgram or -D
Use datagram socket (UDP or unix-dgram) to send the messages to the target. Requires the --inet option as well.
dont-parse or -d
Do not parse the lines read from the input files, send them as received.
--help or -h
Display a brief help message.
--idle-connection <number-of-connections>
Number of idle connections loggen will establish to the destination. Note that loggen will not send any messages on idle connections, but the connection is kept open using keep-alive messages. This option is usable only when using TCP or TLS connections to the destination. See also the --active-connections option. Default value: 0
--inet or -i
Use the TCP (by default) or UDP (when used together with the --dgram option) protocol to send the messages to the target.
--interval <seconds> or -I <seconds>
The number of seconds loggen will run. Default value: 10
Note
When --interval and --number are used together, loggen will send messages until the period set in --interval expires or the amount of messages set in --number is reached, whichever happens first.
--ipv6 or -6
Specify the destination using its IPv6 address. Note that the destination must have a real IPv6 address.
--loop-reading or -l
Read the file specified in --read-file option in loop: loggen will start reading from the beginning of the file when it reaches the end of the file.
--number <number-of-messages> or -n <number-of-messages>
Number of messages to generate.
Note
When --interval and --number are used together, loggen will send messages until the period set in --interval expires or the amount of messages set in --number is reached, whichever happens first.
--no-framing or -F
Do not use the framing of the IETF-syslog protocol style, even if the --syslog-proto option is set.
--quiet or -Q
Display statistics only when loggen is finished. If not set, the statistics are displayed every second.
--permanent or -T
Keep sending logs indefinitely, without time limit.
--rate <message/second> or -r <message/second>
The number of messages generated per second for every active connection. Default value: 1000
If you want to change the message rate while loggen is running, send SIGUSR1 to double the message rate, or SIGUSR2 to halve it:
kill -USR1 <loggen-pid>``kill -USR2 <loggen-pid>
--read-file <filename> or -R <filename>
Read the messages from a file and send them to the target. See also the --skip-tokens option.
Specify - as the input file to read messages from the standard input (stdio). Note that when reading messages from the standard input, loggen can only use a single thread. The -R -parameters must be placed at end of command, like: loggen 127.0.0.1 1061 --read-file -
--sdata <data-to-send> or -p <data-to-send>
Send the argument of the --sdata option as the SDATA part of IETF-syslog (RFC5424 formatted) messages. Use it together with the --syslog-proto option. For example: --sdata "[test name=\\"value\\"]
--size <message-size> or -s <message-size>
The size of a syslog message in bytes. Default value: 256. Minimum value: 127 bytes, maximum value: 8192 bytes.
--skip-tokens <number>
Skip the specified number of space-separated tokens (words) at the beginning of every line. For example, if the messages in the file look like foo bar message, --skip-tokens 2 skips the foo bar part of the line, and sends only the message part. Works only when used together with the --read-file parameter. Default value: 0
--stream or -S
Use a stream socket (TCP or unix-stream) to send the messages to the target.
--syslog-proto or -P
Use the new IETF-syslog message format as specified in RFC5424. By default, loggen uses the legacy BSD-syslog message format (as described in RFC3164). See also the --no-framing option.
--unix </path/to/socket> or -x </path/to/socket>
Use a UNIX domain socket to send the messages to the target.
--use-ssl or -U
Use an SSL-encrypted channel to send the messages to the target. Note that it is not possible to check the certificate of the target, or to perform mutual authentication.
--version or -V
Display version number of syslog-ng.
Examples
The following command generates 100 messages per second for ten minutes, and sends them to port 2010 of the localhost via TCP. Each message is 300 bytes long.
pdbtool — An application to test and convert pattern database rules
Synopsis
pdbtool [command] [options]
Description
The AxoSyslog application can match the contents of the log messages to a database of predefined message patterns (also called patterndb). By comparing the messages to the known patterns, AxoSyslog is able to identify the exact type of the messages, tag the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The functionality of the pattern database is similar to that of the logcheck project, but the syslog-ng approach is faster, scales better, and is much easier to maintain compared to the regular expressions of logcheck.
The pdbtool application is a utility that can be used to:
dump the RADIX tree built from the pattern database (or a part of it) to explore how the pattern matching works.
The dictionary command
dictionary [options]
Lists every name-value pair that can be set by the rules of the pattern database.
--dump-tag or -T
List the tags instead of the names of the name-value pairs.
--pdb <path-to-file> or -p <path-to-file>
Name of the pattern database file to use.
--program <programname> or -P <programname>
List only the name-value pairs that can be set for the messages of the specified ${PROGRAM} application.
The dump command
dump [options]
Display the RADIX tree built from the patterns. This shows how are the patterns represented in AxoSyslog and it might also help to track down pattern-matching problems. The dump utility can dump the tree used for matching the PROGRAM or the MSG parts.
--debug or -d
Enable debug/diagnostic messages on stderr.
--pdb or -p
Name of the pattern database file to use.
--program or -P
Displays the RADIX tree built from the patterns belonging to the ${PROGRAM} application.
Use the match command to test the rules in a pattern database. The command tries to match the specified message against the patterns of the database, evaluates the parsers of the pattern, and also displays which part of the message was parsed successfully. The command returns with a 0 (success) or 1 (no match) return code and displays the following information:
the class assigned to the message (that is, system, violation, and so on),
the ID of the rule that matched the message, and
the values of the parsers (if there were parsers in the matching pattern).
The match command has the following options:
--color-out or -c
Color the terminal output to highlight the part of the message that was successfully parsed.
--debug or -d
Enable debug/diagnostic messages on stderr.
--debug-csv or -C
Print the debugging information returned by the --debug-pattern option as comma-separated values.
--debug-pattern or -D
Print debugging information about the pattern matching. See also the --debug-csv option.
--file=<filename-with-path> or -f
Process the messages of the specified log file with the pattern database. This option allows to classify messages offline, and to apply the pattern database to already existing logfiles. To read the messages from the standard input (stdin), specify a hyphen (-) character instead of a filename.
--filter=<filter-expression> or -F
Print only messages matching the specified AxoSyslog filter expression.
--message or -M
The text of the log message to match (only the ${MESSAGE} part without the syslog headers).
--pdb or -p
Name of the pattern database file to use.
--program or -P
Name of the program to use, as contained in the ${PROGRAM} part of the syslog message.
--template=<template-expression> or -T
An AxoSyslog template expression that is used to format the output messages.
--verbose or -v
Enable verbose messages on stderr.
Example
The following command checks if the patterndb.xml file recognizes the Accepted publickey for myuser from 127.0.0.1 port 59357 ssh6 message:
pdbtool match -p patterndb.xml -P sshd -M "Accepted publickey for myuser from 127.0.0.1 port 59357 ssh6"
Example
The following example applies the sshd.pdb pattern database file to the log messages stored in the /var/log/messages file, and displays only the messages that received a useracct tag.
pdbtool match -p sshd.pdb \
–file /var/log/messages \
–filter ‘tags(“usracct”);’
The merge command
merge [options]
Use the merge command to combine separate pattern database files into a single file (pattern databases are usually stored in separate files per applications to simplify maintenance). If a file uses an older database format, it is automatically updated to the latest format (V3). See the AxoSyslog documentation for details on the different pattern database versions.
--debug or -d
Enable debug/diagnostic messages on stderr.
--directory or -D
The directory that contains the pattern database XML files to be merged.
--glob or -G
Specify filenames to be merged using a glob pattern, for example, using wildcards. For details on glob patterns, see man glob. This pattern is applied only to the filenames, and not on directory names.
--pdb or -p
Name of the output pattern database file.
--recursive or -r
Merge files from subdirectories as well.
--sort or -s
Sort files into alphabetic order during the merge (first sort by filename, then by directory name).
Currently it is not possible to convert a file without merging, so if you only want to convert an older pattern database file to the latest format, you have to copy it into an empty directory.
The patternize command
patternize [options]
Automatically create a pattern database from a log file containing a large number of log messages. The resulting pattern database is printed to the standard output (stdout). The pdbtool patternize command uses a data clustering technique to find similar log messages and replacing the differing parts with @ESTRING:: @ parsers. For details on pattern databases and message parsers, see the AxoSyslog documentation. The patternize command is available only in version 3.2 and later.
--debug or -d
Enable debug/diagnostic messages on stderr.
--file=<path> or -f
The logfile containing the log messages to create patterns from. To receive the log messages from the standard input (stdin), use -.
--iterate-outliers or -o
Recursively iterate on the log lines to cover as many log messages with patterns as possible.
--named-parsers or -n
The number of example log messages to include in the pattern database for every pattern. Default value: 1
--no-parse or -p
Do not parse the input file, treat every line as the message part of a log message.
--samples=<number-of-samples>
Include a generated name in the parsers, for example, .dict.string1, .dict.string2, and so on.
--support=<number> or -S
A pattern is added to the output pattern database if at least the specified percentage of log messages from the input logfile match the pattern. For example, if the input logfile contains 1000 log messages and the --support=3.0 option is used, a pattern is created only if the pattern matches at least 3 percent of the log messages (that is, 30 log messages). If patternize does not create enough patterns, try to decrease the support value.
Use the test command to validate a pattern database XML file. Note that you must have the xmllint application installed. The test command is available only in AxoSyslog version 3.2 and later.
--color-out or -c
Enable coloring in terminal output.
--debug or -d
Enable debug/diagnostic messages on stderr.
--debug or -D
Print debugging information on non-matching patterns.
--rule-id or -r
Test only the patterndb rule (specified by its rule id) against its example.
persist-tool — Display the content of the persist file
Synopsis
persist-tool [command] [options]
Description
The persist-tool application is a utility that can be used to dump the content of the persist file, and manipulate its content.
WARNING: Persist-tool is a special tool for AxoSyslog experts. Do not use the tool unless you know exactly what you are doing. Misconfiguring it will result in irrecoverable damage to the persist file, without any warning.
Limitations
Wildcard characters are not supported in file/directory names.
The dump command
Use the dump command to print the current content of the persist file in JSON format to the console.
Use the add command to add or modify a specified state-entry in the persist file. The state-entry should be in the same format as the dump command displays it. If the given state-entry already exists, it will be updated. Otherwise, a new value will be added. If the given persist state is invalid, it will be skipped.
To use the add command: use persist-tool dump to print the content of the current persist file, and redirect it to a file. Edit the content of this file. Use persist-tool add with this file to modify the persist. The add command has the following options:
--help or -?
Display a brief help message.
--output-dir=<directory> or -o
Required parameter. The directory where the persist file is located at. The name of the persist file stored in this directory must be syslog-ng.persist.
Secure logging is an extension to AxoSyslog providing forward integrity and confidentiality of system logs. It is implemented in form of a module and is configured as a template in the AxoSyslog configuration file.
The main objective of the secure logging module is to provide tamper evident logging, i.e. to adequately protect log records of an information system and to provide a sensor indicating attack attempts. The secure logging module achieves this by authentically encrypting each log record with an individual cryptographic key used only once and protecting the integrity of the whole log archive by a cryptographic authentication code. Each attempt to tamper with either an individual log record or the log archive itself will be immediately detected during log archive verification. Therefore, an attacker can no longer tamper with log records without being detected.
In order to use the log file created by the secure logging module for analysis, the log file must first be decrypted and its integrity verified. This is achieved with a command line utility that is part of the secure logging module and is installed as part of the AxoSyslog package. This utility can easily be integrated into the import module of existing analysis environments.
The secure logging environment uses a cryptographic key for encrypting log entries. Each individual log entry is encrypted with its own key which is immediately discarded after successful encryption in order to provide forward integrity. An efficient algorithm generates the key for the next log entry based the key used for encrypting the previous log entry. The resulting chain of keys preserves forward integrity, i.e. a potential attacker cannot deduce the previous key from the current key.
In order to ease log verification and analysis, a sequence number is added to each log entry. This sequence number is also added to the encryption key creating a one-to-one correspondence of the encryption key with the log entry. In order to prevent truncation attacks or deletion of individual log entries, a message authentication code (MAC) is iteratively applied to the complete log archive and stored in a separate file. It is used upon verification to check the integrity of the whole log archive.
Here is an example of three short original log messages that will be sent to a destination with secure logging enabled:
This is a log message
And here comes another log message
This is a log message with a longer text that is processed without any problems
In order to inspect the status of the secure logging environment, one can check the sequence counter by
querying the key file with the slogkey utility like this:
Counting starts at zero. This is why the counter is set to three after processing three messages.
The output of the secure logging template for the three example messages is shown below. One clearly sees the sequence number that was prepended to each message. The colon indicates the end of the sequence number and the start of the original message. As three message were processed, the sequence counter of the key will be three.
The output of a successful verification run is shown below.
0000000000000000: This is a log message
0000000000000001: And here comes another log message
0000000000000002: This is a log message with a longer text that is processed without any problems
The original log messages have been successfully restored. Additionally, the sequence counter is also prepended to the clear text messages. This helps when in analyzing problems with a particular log entry. As real log files will contain thousands of entries. the sequence counter eases identification of erroneous entries.
Before the secure logging module can be used as part of an existing AxoSyslog installation, several preparatory activities need to be performed.
Key generation
In order to bootstrap the system, an initial key k0 must be created and installed on the log host before secure logging environment is started for the first time.
The newly created host key k0 has its counter set to 0 indicating that it represents the initial host key k0. This host key k0 must be kept secret and not be disclosed to third parties. It will be required to successfully decrypt and verify log archives processed by the secure logging environment. As each log entry will be encrypted with its own key, a new host key will be created after successful processing of a log entry and will replace the previous key. Therefore, the initial host key needs to be stored in a safe place before starting the secure logging environment, as it will be deleted from the log host after processing of the first log entry. The following steps must be done before starting the secure logging environment. Steps 1 and 2 are performed with the slogkey utility. See the slogkey.1 manual page for details on how to generate a master key and to derive a host key from it. Step 3 and 4 depend on the actual deployment in a target environment.
Create a master key
Derive an initial host key k0 from a previously created master key1. Store the initial host key k0 in a safe location outside of the log host
Deploy the key k0 on the log host where the secure logging module will be used
Configuration
Secure logging is configured by adding the corresponding statements to the syslog-ng.conf file.
Secure logging is implemented as a template and is configured accordingly. Apart from the actual template configuration, no other settings are required in order to activate secure logging. The secure logging is activated by placing the following statement in the configuration file.
The name of the secure logging template function. This name can be also be found by calling syslog-ng with the --module-registry arguments and checking the template-func property of the secure logging module in the corresponding output.
--key-file or -k
The host key. <host key file> is the full path of the file storing the host key on the log host. If this arguments is not supplied or does not point to a valid regular key file, syslog-ng will not start and a display a corresponding error message.
--mac-file or -m
The MAC file. <MAC file> is the full path of the MAC file on the log host. The file does not need to exist, as it will be automatically created upon the initial start. If the path is not correct, syslog-ng will not start and a display a corresponding error message.
$RAWMSG
$RAWMSG provides access to the original log message received at the source. This macro is only available if the store-raw-message flag was set for the source. Otherwise, an empty string will be passed to the secure logging template. If access to the original message is not available, for example, if the source does not support the store-raw-message flag, then the $MSG macro can also be used. In this case, however, the integrity guarantee provided by secure logging is limited to the content that this macro provides and does not protect the complete original message.
\n
\n is the line separator. This is important, as the secure logging template expects log entries to be separated by a line separator. When detecting a line separator, the log entry is regarded as complete and is encrypted with the current host key. Therefore, only a single line separator is allowed.
The secure logging template can be combined with any source or destination within the following limitations:
Sources must be line-oriented. Secure logging uses a line separator in order to distinguish between individual log entries. Sources which provide data in a different format, for example, in the form of raw data obtained directly from a database system, cannot currently be used with the secure logging template, as the separation of log entries is not clearly defined for this type of data.
Only sources for which the store-raw-message flag is implemented and set do benefit from the integrity guarantee provided by the secure logging template. Secure logging aims at protecting the integrity of complete log messages including all associated meta-data, such as timestamps and host names. syslog-ng parses the log message into its internal format and provide easy access to parts of a message through macros. While this is convenient when rewriting log messages, it is not helpful for secure logging. syslog-ng provides the store-raw-message flag which provides access to a copy of the original log message after parsing. This is the log message processed and protected by the secure logging template. If the source does not support the store-raw-message flag, then the $MSG macro can also be used. However, in this case the integrity guarantee provided by secure logging is limited to the content that this macro provides.
Log rotation of any kind cannot be used with destinations using secure logging. The reason is that log rotate will overwrite, i.e. delete previous log files. This destroys the cryptographic chain of trust of the log entries making recovery impossible. In order to allow for a an efficient handling of log files, the secure logging environment features iterative verification. Using iterative verification, a log file can be verified in steps. For this to work, the log file must first be downloaded from the log host, together with the corresponding host key and MAC file to a verification host. After this download the log file can be safely deleted from the log host. Verification is then performed on the verification host using the iterative mode of the slogverify utility.
The following example configuration shows the use of the secure logging template on a file destination.
############################################## Minimal syslog-ng.conf file with secure logging enabled. Encrypted log# entries will be logged to a single file called /var/log/messages.slog#@version: 4.4
@include "scl.conf"source s_local { system(); internal();};source s_network { network( transport("udp") port(514) flags(store-raw-message));};# Secure logging template definitiontemplate secure_logging { template("$(slog --key-file /etc/syslog-ng/host.key --mac-file /etc/syslog-ng/mac.dat $RAWMSG)\n");};# This configures a secure logging destinationdestination d_local { file("/var/log/messages.slog" template(secure_logging));};log { source(s_local);# This source has the raw message flag set source(s_network);# This statement activates secure logging for this destination destination(d_local);};
Log Verification
In order to analyze the log file created in a secure logging environment, the log files must be decrypted and their integrity be verified. Verification requires both the initial host key k0 and the corresponding MAC file and is performed with the slogverify utility. It is not normally performed on the log host where the secure logging environment is producing log data. In a typical deployment, log files would be retrieved from the log host and transferred to a central log collector where verification it performed. As verification requires the use of the initial host key k0, it should only be performed in a trusted environment.
Normal mode
In normal mode, a complete log archive is verified at once. In a typical environment, this would mean retrieving a log file together with is MAC file from a log host and retrieving the corresponding initial key k0 form a safe location and supplying them to the slogverify utility. A typical call sequence for verification in normal mode would look like this:
The initial host key k0. Supplying the initial key k0 is enough for decrypting all log entries, as the key derivation algorithm is able to generate the necessary keys for all subsequent log entries based on the initial key k0.
mac.dat
The MAC file from the log host.
/var/log/messages.slog
The file containing the encrypted log entries as retrieved from a log host.
/var/log/verified/messages
The file receiving the plain text log after decryption.
Log files may become large and not fit into system memory. Verification is therefore performed in chunks. Each part of the log file is transferred to an internal buffer on which verification is performed. After the buffer has been processed, the next chunk is fetched. An optional buffer argument can be supplied to the slogverify utility in order to change the default buffer size of 1000 log entries to a number suitable for the system on which the verification is performed, for example:
See slogverify for details on verification in normal mode.
Iterative mode
Verification in normal mode may not be suitable for some application scenarios. Many log hosts use log rotation in order to preserve storage space. In log rotation, a threshold for the maximum amount of storage space and the number of generations is defined for different type of log files. When either storage space is exhausted or the number of generations is reached, the oldest log file will be overwritten by new incoming log data. This procedure is not possible in secure logging environment, as overwriting, i.e. deleting a log file would break the cryptographic chain that is established between the log entries. This comes as no surprise, as one of the main objectives of secure logging is to protect against deletion of log entries by a potential attacker.
In order allow for a procedure similar to log rotation, the secure logging environment features an iterative mode. In iterative mode, log files can be split into different files and each of these files can be verified separately. Care must be taken when performing verification in iterative mode, as each of the different log files needs to be accompanied by a copy of the host key and the MAC files present on the system at the time of retrieval. A typical usage scenario for the iterative mode would look like this:
Define a storage threshold for the secure logging file destination. In this example we assume 500MB.
Let the secure logging environment produce log data that is written to the destination until 500MB are reached.
Stop the secure logging environment and retrieve the log file, the host key and the MAC files from the log host.
Delete the log file on the log host but leave host key and MAC file untouched.
Restart the secure logging environment.
Perform verification in iterative mode with the log file, the host key and the MAC just retrieved.
Steps 2-6 have to repeated each time the log file reaches a size of 50 MB. Assuming that the log file parts will be named after the iteration, e.g. log.1, log.2, log.3, etc. and a similar convention is applied to the host keys and MAC files, a typical call sequence for the validation of a log file part in iterative mode after three iterations will look like this:
The host key from the previous iteration. In this example, this is the second iteration.
mac.dat.2
The MAC file from the previous iteration. In the example, verification is performed during the third iteration, so the MAC file from the second iteration is required.
mac.dat
The current MAC file from the log host.
/var/log/messages.slog.3
The file part containing the encrypted log entries as retrieved from the log host during the third iteration.
/var/log/verified/messages.3
The file receiving the plain text log after decryption during the third iteration.
In a real deployment, the above steps would typically be automated using a scripting engine. See slogverify(1) for details on verification in iterative mode.
slogencrypt — Encrypt existing plain text log files using the AxoSyslog secure logging environment.
Synopsis
slogencrypt [options] [arguments]
Description
The slogencrypt utility is used to encrypt plain text log file using an existing secure logging environment. Using this utility, log files obtained from a previous installation of syslog-ng or another logging system can be transferred to a secure logging environment. The order of the log entries is preserved. Encrypting plain text log files using an existing secure logging environment, requires the current encryption key to be supplied in order to preserve consistency.
General call sequence: slogencrypt -k <key file> -m <MAC file> <new key file> <new MAC file> <plain text log> <output file> [counter]
Arguments
<new key file>
The file that will contain the new current encryption key after successful encryption.
<new MAC file>
The file receiving the new current message authentication code (MAC) of the secure encrypted destination after encryption. In case an existing file is supplied, new entries will be appended.
<input log file>
The plain text log file that will be encrypted using the secure logging environment.
<output log file>
The file that will contain the encrypted log entries from the supplied plain text log file after encryption.
counter
The current log entry counter of the secure encrypted destination after encryption. This is required if the log entries to be encrypted will be appended to an existing secure encrypted destination.
Options
--key-file or -k
The current host key from the system where the encryption will be performed.
--mac-file or -m
The current MAC file from the system where the encryption will be performed.
slogkey — Manage cryptographic keys for use with the AxoSyslog secure logging environment.
Synopsis
slogkey [options] [arguments]
Description
The slogkey utility is used to manage cryptographic keys for use with the secure logging module of AxoSyslog. Use this utility to create a master key, derive a host key to be used by a secure logging configuration and to display the current sequence counter of a key. The options determine the operating mode and are mutually exclusive.
<filename>: The name of the file to which the master key will be written.
Host key derivation
Call sequence: slogkey --derive-key <master key file> <host MAC address> <host serial number> <host key file>
<master key file>: The master key from which the host key will be derived.
<host MAC address>: The MAC address of the host on which the key will be used. Instead of the MAC address, any other string that uniquely identifies a host can be supplied, e.g. the company inventory number.
<host serial number>: The serial number of the host on which the key will be used. Instead of the serial number, any other string that uniquely identifies a host can be supplied, for example, the company inventory number.
<host key file>: The name of the file to which the host key will be written.
NOTE: The newly created host key has its counter set to 0 indicating that it represents the initial host key k0. This host key must be kept secret and not be disclosed to third parties. It will be required to successfully decrypt and verify log archives processed by the secure logging environment. As each log entry will be encrypted with its own key, a new host key will be created after successful processing of a log entry and will replace the previous key. Therefore, the initial host key needs to be stored in a safe place before starting the secure logging environment, as it will be deleted from the log host after processing of the first log entry.
The slogverify utility is used to verify the integrity of cryptographically secured logs and to decrypt log entries produced in a AxoSyslog secure logging environment.
Normal mode: slogverify -k <host key file> -m <input MAC file> <input file> <output file> [buffers]
An encrypted log file from the syslog-ng secure logging environment that will be verified.
output file
The file that will contain the plain text log entries after decryption and verification.
buffers
Optional number of input buffers. The number of buffers can be used for performance adjustments in case the log file to be verified is very large and cannot be processed at once. It is a positive number of log entries that can be held in memory during verification. The minimum number if 10 and the maximum number is 4294967295. If this argument is not supplied the default of 1000 is used.
Options
--iterative or -i
Iterative mode. This is useful in case the log files are periodically copied from the system on which they where generated to central collector. As log rotation, i.e. overwriting log files in order to preserve space cannot be done in a secure logging environment, the iterative mode can be used instead. This works as follows: If a certain storage limit is reached the log file together with the host key and the MAC file is copied to new destination and the old file is deleted. The verification is then performed in iterations, i.e. separately for each file that was retrieved from the log host. For this to work, it is important to always retrieve the corresponding host key and MAC files. The process can be automated, e.g. by calling slogverify in iterative mode from a script.<
--key-file or -k
The initial host key (k0). This option is used in normal mode only.
--mac-file or -m
The current MAC file used.
--prev-key-file or -p
The host key corresponding to the previous log file. This option can be used in iterative mode only. In theory, this can be initial host key (k0) but using this key might generate warnings, as the gap between the first log entry ever (log entry 0) and the first log entry of the current log file might be large.
--prev-mac-file or -r
The MAC file from the previous log file. This option can only be used in iterative mode.
The syslog-debun application is distributed with the AxoSyslog system logging application, and is usually part of the AxoSyslog package.
The syslog-debun tool collects and saves information about your AxoSyslog installation, making troubleshooting easier, especially if you ask help about your AxoSyslog related problem.
General Options
-r
Run syslog-ng-debun. Using this option is required to actually execute the data collection with syslog-ng-debun. It is needed to prevent accidentally running syslog-ng-debun.
-h
Display the help page.
-l
Do not collect privacy-sensitive data, for example, process tree, fstab, and so on. If you use with -d, then the following parameters will be used for debug mode:-Fev
-R <directory>
The directory where AxoSyslog is installed instead of /opt/syslog-ng.
-W <directory>
Set the working directory, where the debug bundle will be saved. Default value: /tmp. The name of the created file is syslog.debun.${host}.${date}.${3-random-characters-or-pid}.tgz
Debug mode options
-d
Start AxoSyslog in debug mode, using the -Fedv --enable-core options.
Warning
Using this option under high message load may increase disk I/O during the debug, and the resulting debug bundle can be huge. To exit debug mode, press Enter.
-D <options>
Start AxoSyslog in debug mode, using the specified command-line options. To exit debug mode, press Enter.
-t <seconds>
Run AxoSyslog in non-interactive debug mode for <seconds>, and automatically exit debug mode after the specified number of seconds.
-w <seconds>
Wait <seconds> seconds before starting debug mode.
System call tracing
-s
Enable syscall tracing (strace -f or truss -f). Note that using -s itself does not enable debug mode, only traces the system calls of an already running AxoSyslog process. To trace system calls in debug mode, use both the -s and -d options.
Packet capture options
Capturing packets requires a packet capture tool on the host. The syslog-debun tool attempts to use tcpdump on most platforms, except for Solaris, where it uses snoop.
-i <interface>
Capture packets only on the specified interface, for example, eth0.
-p
Capture incoming packets using the following filter: port 514 or port 601 or port 53
-P <options>
Capture incoming packets using the specified filter.
-t <seconds>
Run AxoSyslog in non-interactive debug mode for <seconds>, and automatically exit debug mode after the specified number of seconds.
Examples
syslog-ng-debun -r
Create a simple debug bundle, collecting information about your environment, for example, list packages containing the word: syslog, ldd of your syslog-binary, and so on.
syslog-ng-debun -r -l
Similar to syslog-ng-debun -r, but without privacy-sensitive information. For example, the following is NOT collected: fstab, df output, mount info, ip / network interface configuration, DNS resolve info, and process tree.
syslog-ng-debun -r -d
Similar to syslog-ng-debun -r, but it also stops AxoSyslog, then restarts it in debug mode (-Fedv --enable-core). To stop debug mode, press Enter. The output of the debug mode collected into a separate file, and also added to the debug bundle.
syslog-ng-debun -r -s
Trace the system calls (using strace or truss) of an already running AxoSyslog process.
syslog-ng-debun -r -d -s
Restart AxoSyslog in debug mode, and also trace the system calls (using strace or truss) of the AxoSyslog process.
syslog-ng-debun -r -p
Run packet capture (pcap) with the filter: port 514 or port 601 or port 53 Also waits for pressing Enter, like debug mode.
syslog-ng-debun -r -p -t 10
Noninteractive debug mode: Similar to syslog-ng-debun -r -p, but automatically exit after 10 seconds.
Change the packet-capturing filter from the default to host 1.2.3.4. Also change debugging parameters from the default to -Fev --enable-core. Since a timeout (-t) is not given, waits for pressing Enter.
syslog-ng-debun -r -p -d -w 5 -t 10
Collect pcap and debug mode output following this scenario:
Start packet capture with default parameters (-p)
Wait 5 seconds (-w 5)
Stop AxoSyslog
Start AxoSyslog in debug mode with default parameters (-d)
syslog-ng-ctl — Display message statistics and enable verbose, debug and trace modes
Synopsis
syslog-ng-ctl [command] [options]
Description
Note
The syslog-ng-ctl application is distributed with the AxoSyslog system logging application, and is usually part of the AxoSyslog package.
The syslog-ng-ctl application is a utility that can be used to:
enable/disable various AxoSyslog messages for troubleshooting
display statistics about the processed messages
handling password-protected private keys
display the currently running configuration of AxoSyslog
reload the configuration of AxoSyslog.
Enabling troubleshooting messages
syslog-ng-ctl log-level <level>
Available in AxoSyslog 4.0 and later.
Use the syslog-ng-ctl log-level <level> command to display verbose, trace, or debug messages. If you are trying to solve configuration problems, the verbose (and occasionally debug) messages are usually sufficient. Trace messages are needed mostly for finding software errors. After solving the problem, do not forget to return the log level to the default using the syslog-ng-ctl log-level default command.
Use syslog-ng-ctl log-level without any parameters to display the current log level.
If AxoSyslog was started with the --stderr or -e option, the messages will be sent to stderr. If not specified, AxoSyslog will log such messages to its internal source.
If you need to use a non-standard control socket to access syslog-ng, use the syslog-ng-ctl <command> --control=<socket> command to specify the socket to use.
Higher log-levels automatically include messages from lower log-levels:
default: Just normal log messages.
verbose: Normal and verbose log messages.
debug: Include debug messages of AxoSyslog.
trace: Include trace messages of how messages are processed.
Example
syslog-ng-ctl log-level verbose
syslog-ng-ctl query
The AxoSyslog application stores various data, metrics, and statistics in a hash table. Every property has a name and a value. For example:
You can query the nodes of this tree, and also use filters to select the information you need. A query is actually a path in the tree. You can also use the ? and \* wildcards. For example:
Select every property: \*
Select all dropped value from every stats node: \*.stats.dropped
The nodes and properties available in the tree depend on your AxoSyslog configuration (that is, the sources, destinations, and other objects you have configured), and also on your stats-level() settings.
The list command
syslog-ng-ctl query list
Use the syslog-ng-ctl query list command to display the list of metrics that AxoSyslog collects about the processed messages.
The syslog-ng-ctl query list command has the following options:
--reset
Use --reset to set the selected counters to 0 after executing the query, except for the queued and the memory_usage counters. (The queued counters show the number of messages in the message queue of the destination driver, waiting to be sent to the destination. The memory_usage counters show the amount of memory used by the messages in the different queue types (in bytes). This includes every queue used by the object, including memory buffers (log-fifo) and disk-based buffers (both reliable and non-reliable))
Displaying metrics and statistics
syslog-ng-ctl query get [options]
The syslog-ng-ctl query get <query> command lists the nodes that match the query, and their values.
For example, the destination query lists the configured destinations, and the metrics related to each destination. An example output:
The syslog-ng-ctl query get command has the following options:
--sum
Add up the result of each matching node and return only a single number.
For example, the syslog-ng-ctl query get --sum "destination\*.dropped" command displays the number of messages dropped by the AxoSyslog instance.
--reset
Use --reset to set the selected counters to 0 after executing the query, except for the queued and the memory_usage counters. (The queued counters show the number of messages in the message queue of the destination driver, waiting to be sent to the destination. The memory_usage counters show the amount of memory used by the messages in the different queue types (in bytes). This includes every queue used by the object, including memory buffers (log-fifo) and disk-based buffers (both reliable and non-reliable))
The stats command
stats [options]
Use the stats command to display statistics about the processed messages. For details about the displayed statistics, see The AxoSyslog documentation. The stats command has the following options:
--control=<socket> or -c
Specify the socket to use to access AxoSyslog. Only needed when using a non-standard socket.
--reset=<socket> or -r
Reset all statistics to zero, except for the queued and the memory_usage counters. (The queued counters show the number of messages in the message queue of the destination driver, waiting to be sent to the destination. The memory_usage counters show the amount of memory used by the messages in the different queue types (in bytes). This includes every queue used by the object, including memory buffers (log-fifo) and disk-based buffers (both reliable and non-reliable))
--remove-orphans
Safely removes all counters that are not referenced by any syslog-ng stat producer objects.
The flag can be used to prune dynamic and static counters manually. This is useful, for example, when a templated file destination produces a lot of stats:
The stats-lifetime() can be used to do the same automatically and periodically, but currently stats-lifetime() removes only dynamic counters that have a timestamp field set.
The syslog-ng-ctl credentials status command allows you to query the status of the private keys that AxoSyslog uses in the network() and syslog() drivers. You can also provide the passphrase for password-protected private keys using the syslog-ng-ctl credentials add command. For details on using password-protected keys, see The syslog-ng Administrator Guide.
Displaying the status of private keys
syslog-ng-ctl credentials status [options]
The syslog-ng-ctl credentials status command allows you to query the status of the private keys that AxoSyslog uses in the network() and syslog() drivers. The command returns the list of private keys used, and their status. For example:
syslog-ng-ctl credentials status
Secret store status:
/home/user/ssl_test/client-1/client-encrypted.key SUCCESS
If the status of a key is PENDING, you must provide the passphrase for the key, otherwise AxoSyslog cannot use it. The sources and destinations that use these keys will not work until you provide the passwords. Other parts of the AxoSyslog configuration will be unaffected. You must provide the passphrase of the password-protected keys every time AxoSyslog is restarted.
The following log message also notifies you of PENDING passphrases:
Waiting for password;keyfile='private.key'
--control=<socket> or -c
Specify the socket to use to access AxoSyslog. Only needed when using a non-standard socket.
Opening password-protected private keys
syslog-ng-ctl credentials add [options]
You can add the passphrase to a password-protected private key file using the following command. AxoSyslog will display a prompt for you to enter the passphrase. We recommend that you use this method.
Specify the socket to use to access AxoSyslog. Only needed when using a non-standard socket.
--id=<path-to-the-key> or -i
The path to the password-protected private key file. This is the same path that you use in the key-file() option of the AxoSyslog configuration file.
--secret=<passphrase-of-the-key> or -s
The password or passphrase of the private key.
Displaying the configuration
syslog-ng-ctl config [options]
Use the syslog-ng-ctl config command to display the configuration that AxoSyslog is currently running. By default, only the content of the main configuration file is displayed, included files are not resolved. To resolve included files and display the entire configuration, use the syslog-ng-ctl config --preprocessed command.
Starting with AxoSyslog version 4.2, you can display the configuration identifier (if set) and the SHA256 has of the output of the syslog-ng-ctl config --preprocessed command by running syslog-ng-ctl config --id. For details, see Configuration identifier.
Reloading the configuration
syslog-ng-ctl reload [options]
Use the syslog-ng-ctl reload command to reload the configuration file of AxoSyslog without having to restart the AxoSyslog application. The syslog-ng-ctl reload works like a SIGHUP.
The syslog-ng-ctl reload command returns 0 if the operation was successful, 1 otherwise.
The healthcheck command
Available in AxoSyslog 4.2 and later.
You can use the syslog-ng-ctl healthcheck command to query the healthcheck status of AxoSyslog. The following health values are reported:
mainloop_io_worker_roundtrip_latency_nanoseconds: mainloop->io-worker-job->mainloop roundtrip - a basic latency measure for AxoSyslog.
syslogng_internal_events_queue_usage_ratio: If you are using the internal() source in your configuration, then this value shows the saturation of the internal source’s queue, ranging from 0 to 1. Non-zero values indicate some kind of disruption in the pipelines.
You can run syslog-ng-ctl healthcheck --timeout <seconds> to use as a boolean healthy/unhealthy check.
Health checks are also published as periodically updated metrics. You can configure the frequency of these checks with the stats(healthcheck-freq()) option. The default is 5 minutes.
Permission is hereby granted, free of charge, to any person obtaining a copy of these documentation files (the “Documentation”), to use the Documentation subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or portions of the Documentation. Any and all copies of the above copyright and this permission notice contained in the Documentation shall not be removed, obscured, or modified.
If you modify the Documentation, You must cause any modified files to carry prominent notices stating that You changed the files.
The Documentation contains trademarks and registered trademarks owned by One Identity and its affiliates. Permission is granted to make and distribute verbatim copies of the Documentation from the official syslog-ng.com site. Modified versions of the Documentation containing One Identity trademarks are permitted exclusively for internal use; however, all external distributions of modified Documentation are prohibited from containing any One Identity trademarks which include syslog-ng, the syslog-ng logo, Balabit, the Balabit logo, One Identity and the One Identity logo.
No relationship is established as a result of this Documentation license. Any statements that suggest a partnership or other relationship with One Identity and its affiliates are prohibited.
THE DOCUMENTATION IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE DOCUMENTATION OR THE USE OR OTHER DEALINGS IN THE DOCUMENTATION.
YOU ACKNOWLEDGE THAT THE DOCUMENTATION YOU ARE DOWNLOADING IS SUBJECT TO THE RESTRICTIONS AND CONTROLS IMPOSED BY UNITED STATES EXPORT REGULATIONS.
YOU CERTIFY THAT:
YOU DO NOT INTEND TO USE THE DOCUMENTATION FOR ANY PURPOSE PROHIBITED BY UNITED STATES EXPORT REGULATIONS, INCLUDING, WITHOUT LIMITATION, TERRORISM, CYBER-ATTACKS, CYBER-CRIMES, MONEY-LAUNDERING, INDUSTRIAL ESPIONAGE, OR NUCLEAR, CHEMICAL OR BIOLOGICAL WEAPONS PROLIFERATION.
YOU ARE NOT LISTED AS A DENIED PARTY ON ANY LIST GOVERNING UNITED STATES EXPORTS.
YOU ARE NOT A NATIONAL OF ANY COUNTRY THAT IS NOT APPROVED TO RECEIVE A U.S. EXPORT. AS OF 2023, THESE COUNTRIES ARE CUBA, IRAN, NORTH KOREA, SYRIA, BELARUS AND RUSSIA.
If any of the above applicable conditions are not met or if your certification is inaccurate, this restricted copyright license is immediately terminated.
