This chapter provides a very brief introduction into configuring the AxoSyslog application. For details on the format of the configuration file and how to configure sources, destinations, and other features, refer to the subsequent chapters.
To configure AxoSyslog as a relay that receives log messages from client hosts and forwards them to a central logserver, see Configuring AxoSyslog as a logserver.
To configure AxoSyslog on a client host, complete the following steps.
Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.
Configure the local sources to collect the log messages of the host. Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.
To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.
Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog Helm chart.
Add sources to collect the messages from your log files. File sources look like this:
Note
Many applications send log messages to logfiles by default (for example, the Roundcube webmail client, or the ProFTPD FTP server), but can be configured to send them to syslog instead. If possible, it is recommended to reconfigure the application that way.
Note
The default configuration file of AxoSyslog collects platform-specific log messages and the internal log messages of AxoSyslog.
Terminal window
source s_local { system(); internal();};
Create a network destination that points directly to the AxoSyslog server, or to a local relay. The network destination greatly depends on the protocol that your log server or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:
When both nodes run AxoSyslog, you can use the OpenTelemetry protocol with the axosyslog-otlp() destination instead. It preserves the internal representation of the messages, and adds scalability (via the workers() option), application-layer acknowledgement, and improved load balancing:
Create a log statement connecting the local sources to the file destination.
Note
The default configuration of AxoSyslog has only one log statement:
Terminal window
log { source(s_local); destination(d_local);};
Set filters, macros and other features and options (for example, TLS encryption) as necessary.
Example: The default configuration file of AxoSyslog
The following is the default configuration file of AxoSyslog. It collects local log messages and the log messages of AxoSyslog and saves them in the /var/log/messages file.
The following is a simple configuration file that collects local log messages and forwards them to another AxoSyslog node using the OpenTelemetry protocol.
To configure AxoSyslog on a server host, complete the following steps.
Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.
AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.
To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.
Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.
Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog Helm chart.
Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:
If your clients are also AxoSyslog nodes, you can receive their messages over the OpenTelemetry protocol with the axosyslog-otlp() source. This preserves the internal representation of the messages, and adds scalability, application-layer acknowledgement, and improved load balancing:
Note
Starting with AxoSyslog version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164) and IETF-syslog (RFC 5424-26) messages.
Create local destinations that will store the log messages, for example, file- or program destinations. The default configuration of AxoSyslog places the collected messages into the /var/log/messages file:
Set filters, options (for example, TLS encryption) and other advanced features as necessary.
Note
By default, the AxoSyslog server treats the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the AxoSyslog server, use the keep-hostname(yes) option both on the AxoSyslog relay and the AxoSyslog server. This option can be set individually for every source if needed.
If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.
Example: A simple configuration for servers
The following is a simple configuration file for AxoSyslog that collects incoming log messages and stores them in a text file.
This section describes how to configure AxoSyslog as a relay.
3.1 - Configuring AxoSyslog on relay hosts
To configure AxoSyslog on a relay host, complete the following steps:
Install the AxoSyslog application on the host. For details on installing AxoSyslog on specific operating systems, see Install AxoSyslog.
Configure the network sources that collect the log messages sent by the clients.
Create a network destination that points to the AxoSyslog server.
Note
If both your clients (or the relay) and the upstream server run AxoSyslog, you can use the OpenTelemetry protocol on both legs instead of the syslog protocol: receive messages with an axosyslog-otlp() source and forward them with an axosyslog-otlp() destination. The axosyslog-otlp() driver is available in AxoSyslog version 4.12 and later, and requires the axosyslog-grpc (or axosyslog-mod-grpc) package. For details, see axosyslog-otlp(): Receive logs from another node using OpenTelemetry and axosyslog-otlp(): Forward logs to another node using OpenTelemetry.
Create a log statement connecting the network sources to the AxoSyslog server.
Configure the local sources that collect the log messages of the relay host.
Create a log statement connecting the local sources to the AxoSyslog server.
Enable the keep-hostname() and disable the chain-hostnames() options. (For details on how these options work, see Global options.)
Note
It is recommended to use these options on your AxoSyslog server as well.
Set filters and options (for example, TLS encryption) as necessary.
Note
By default, the AxoSyslog server treats the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the AxoSyslog server, use the keep-hostname(yes) option both on the AxoSyslog relay and the AxoSyslog server. This option can be set individually for every source if needed.
If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.
Example: A simple configuration for relays
The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.
Example: A relay configuration using OpenTelemetry
The following relay configuration receives log messages from AxoSyslog clients over the OpenTelemetry protocol and forwards them to the upstream AxoSyslog server using the same protocol.
Depending on your exact needs about relaying log messages, there are many scenarios and AxoSyslog options that influence how the log message will look like on the logserver. Some of the most common cases are summarized in the following example:
Consider the following example: client-host > relay > server, where the IP address of client-host is 192.168.1.2. The client-host device sends a syslog message to relay. Depending on the settings of relay, the following can happen.
By default, the keep-hostname() option is disabled, so relay writes the IP address of the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discarding any IP address or hostname that was originally in the message.
If the keep-hostname() option is enabled on relay, but name resolution is disabled (the use-dns() option is set to no), relay uses the HOST field of the message as-is, which is probably 192.168.1.2.
To resolve the 192.168.1.2 IP address to a hostname on relay using a DNS server, use the keep-hostname(no) and use-dns(yes) options. If the DNS server is properly configured and reverse DNS lookup is available for the 192.168.1.2 address, AxoSyslog will rewrite the HOST field of the log message to client-host.
Note
It is also possible to resolve IP addresses locally, without relying on the DNS server. For details on local name resolution, see Resolving hostnames locally.
The above points apply to the AxoSyslog server (server) as well, so if relay is configured properly, use the keep-hostname(yes) option on server to retain the proper HOST field. Setting keep-hostname(no) on server would result in AxoSyslog rewriting the HOST field to the address of the host that sent the message to server, which is relay in this case.
If you cannot or do not want to resolve the 192.168.1.2 IP address on relay, but want to store your log messages on server using the IP address of the original host (that is, client-host), you can enable the spoof-source() option on relay. However, spoof-source() works only under the following conditions:
The syslog-ng binary has been compiled with the --enable-spoof-source option.
The log messages are sent using the highly unreliable UDP transport protocol. (Extremely unrecommended.)
4 - Sending Kubernetes logs to OpenSearch
The following tutorial shows you how to install AxoSyslog on Kubernetes, deploy OpenSearch to the same cluster, and send Kubernetes logs to OpenSearch.
Prerequisites
You need:
A Kubernetes cluster. We used minikube with docker driver and Helm. We used a Ubuntu 22.04 (amd64) machine, but it should work on any system that can run minikube (2 CPUs, 2GB of free memory, 20GB of free disk space).
The OpenSearch service needs a large mmap count setting, so set it to at least 262144, for example:
If you don’t already have an application that generates logs deployed to the Kubernetes cluster, install kube-logging/log-generator to generate sample logs. Complete the following steps.
"kube-logging" has been added to your repositories
Update your repositories.
Terminal window
helm repo update
The output should look like:
Terminal window
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kube-logging" chart repository
Update Complete. ⎈Happy Helming!⎈
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "opensearch" chart repository
Update Complete. ⎈Happy Helming!⎈
Install OpenSearch. This step can take a few minutes.
NAME: opensearch-dashboards-1684695728
LAST DEPLOYED: Sun May 21 21:02:09 2023NAMESPACE: default
STATUS: deployed
REVISION: 1TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
exportPOD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch-dashboards-1684695728" -o jsonpath="{.items[0].metadata.name}")exportCONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")echo"Visit http://127.0.0.1:8080 to use your application" kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
Now you should have 5 pods. Check that they exist:
Terminal window
kubectl get pods
The output should look like:
Terminal window
NAME READY STATUS RESTARTS AGE
log-generator-1681984863-5946c559b9-ftrrn 1/1 Running 0 3m39s
opensearch-cluster-master-0 1/1 Running 0 81s
opensearch-cluster-master-1 1/1 Running 0 81s
opensearch-cluster-master-2 1/1 Running 0 81s
opensearch-dashboards-1681999620-59f64f98f7-bjwwh 1/1 Running 0 44s
Forward the 5601 port of the OpenSearch Dashboards service (replace the name of the pod with your pod).
Forwarding from 127.0.0.1:8080 -> 5601Forwarding from [::1]:8080 -> 5601
Log in to the dashboard at http://localhost:8080 with admin/admin. You will soon create an Index Pattern here, but first you have to send some logs from syslog-ng.
Create a YAML file (called axoflow-demo.yaml in the examples) to configure the collector.
config:sources:kubernetes:# Collect kubernetes logsenabled:truedestinations:# Send logs to OpenSearchopensearch:- address:"opensearch-cluster-master"index:"test-axoflow-index"user:"admin"password:"admin"tls:# Do not validate the server's TLS certificate.peerVerify:false# Send the syslog fields + the metadata from .k8s.* in JSON formattemplate:"$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))"
Check how the syslog-ng.conf file looks with your custom values:
Open OpenSearch dashboard at http://localhost:8080/app/management/opensearch-dashboards/.
Create an Index Pattern called test-axoflow-index: http://localhost:8080/app/management/opensearch-dashboards/indexPatterns. At Step 2, set the Time field to @timestamp.
Now you can see your logs on the Discover view at http://localhost:8080/app/discover. Opening the detailed view for a log entry shows you the fields sent to OpenSearch.
5 - Managing and checking the syslog-ng service on Linux
New to AxoSyslog? AxoSyslog is a binary compatible syslog-ng replacement, from the original creator, developed by the same team.
Same architecture, same config files, same paths. Cloud-native images, fast releases, modern observability (OTel, K8s), and powerful data processing: read more about the differences between AxoSyslog and syslog-ng.
Job for syslog-ng.service failed because the control process exited with error code. See `systemctl status syslog-ng.service` and `journalctl -xe`for details.