log: Filter and route log messages using log paths, flags, and filters

• 1: Log paths
• 1.1: Embedded log statements
• 1.1.1: Using embedded log statements
• 1.2: if-else-elif: Conditional expressions
• 1.3: Junctions and channels
• 1.4: Log path flags
• 2: Managing incoming and outgoing messages with flow-control
• 2.1: Flow-control and multiple destinations
• 2.2: Configuring flow-control
• 3: Using disk-based and memory buffering
• 3.1: Enabling reliable disk-based buffering
• 3.2: Enabling normal disk-based buffering
• 3.3: How to get information about disk-buffer files
• 3.3.1: Information about disk-buffer files
• 3.3.2: Getting the status information of disk-buffer files
• 3.3.3: Getting the list of disk-buffer files
• 3.3.4: Printing the content of disk-buffer files
• 3.3.5: Orphan disk-buffer files
• 3.3.6: How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance
• 3.4: Enabling memory buffering
• 3.5: About disk queue files
• 4: Filters
• 4.1: Using filters
• 4.2: Combining filters with boolean operators
• 4.3: Comparing macro values in filters
• 4.4: Using wildcards, special characters, and regular expressions in filters
• 4.5: Tagging messages
• 4.6: Filter functions
• 4.6.1: facility()
• 4.6.2: filter()
• 4.6.3: host()
• 4.6.4: in-list()
• 4.6.5: level() or priority()
• 4.6.6: match()
• 4.6.7: message()
• 4.6.8: netmask()
• 4.6.9: netmask6()
• 4.6.10: program()
• 4.6.11: rate-limit()
• 4.6.11.1: Options of rate-limit() filter
• 4.6.12: source()
• 4.6.13: tags()
• 5: Dropping messages

1 - Log paths

Log paths determine what happens with the incoming log messages. Messages coming from the sources listed in the log statement and matching all the filters are sent to the listed destinations.

To define a log path, add a log statement to the syslog-ng configuration file using the following syntax:

Declaration

   log {
        source(s1); source(s2); ...
        optional_element(filter1|parser1|rewrite1);
        optional_element(filter2|parser2|rewrite2);
        ...
        destination(d1); destination(d2); ...
        flags(flag1[, flag2...]);
    };

Named log paths and log path metrics

In AxoSyslog version 4.1 and later, you can add an ID or name to the log path to make the configuration file more readable. Also, AxoSyslog collects ingress and egress metrics for named log paths. For example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Example: A simple log statement

The following log statement sends all messages arriving to the localhost to a remote server.

   source s_localhost {
        network(
            ip(127.0.0.1)
            port(1999)
        );
    };
    destination d_tcp {
        network("10.1.2.3"
            port(1999)
            localport(999)
        );
    };
    log {
        source(s_localhost);
        destination(d_tcp);
    };

All matching log statements are processed by default, and the messages are sent to every matching destination by default. So a single log message might be sent to the same destination several times, provided the destination is listed in several log statements, and it can be also sent to several different destinations.

This default behavior can be changed using the flags() parameter. Flags apply to individual log paths, they are not global options. For details and examples on the available flags, see Log path flags. The effect and use of the flow-control flag is detailed in Managing incoming and outgoing messages with flow-control.

1.1 - Embedded log statements

Starting from version 3.0, AxoSyslog can handle embedded log statements (also called log pipes). Embedded log statements are useful for creating complex, multi-level log paths with several destinations and use filters, parsers, and rewrite rules.

For example, if you want to filter your incoming messages based on the facility parameter, and then use further filters to send messages arriving from different hosts to different destinations, you would use embedded log statements.

Embedded log statements include sources — and usually filters, parsers, rewrite rules, or destinations — and other log statements that can include filters, parsers, rewrite rules, and destinations. The following rules apply to embedded log statements:

• Only the beginning (also called top-level) log statement can include sources.

• Embedded log statements can include multiple log statements on the same level (that is, a top-level log statement can include two or more log statements).

• Embedded log statements can include several levels of log statements (that is, a top-level log statement can include a log statement that includes another log statement, and so on).

• After an embedded log statement, you can write either another log statement, or the flags() option of the original log statement. You cannot use filters or other configuration objects. This also means that flags (except for the flow-control flag) apply to the entire log statement, you cannot use them only for the embedded log statement.

• Embedded log statements that are on the same level receive the same messages from the higher-level log statement. For example, if the top-level log statement includes a filter, the lower-level log statements receive only the messages that pass the filter.

Embedded log filters can be used to optimize the processing of log messages, for example, to re-use the results of filtering and rewriting operations.

1.1.1 - Using embedded log statements

Embedded log statements (for details, see Embedded log statements) re-use the results of processing messages (for example, the results of filtering or rewriting) to create complex log paths. Embedded log statements use the same syntax as regular log statements, but they cannot contain additional sources. To define embedded log statements, use the following syntax:

   log {
        source(s1); source(s2); ...
    
        optional_element(filter1|parser1|rewrite1);
        optional_element(filter2|parser2|rewrite2);
        ...
        destination(d1); destination(d2); ...
    
        #embedded log statement
        log {
            optional_element(filter1|parser1|rewrite1);
            optional_element(filter2|parser2|rewrite2);
            ...
            destination(d1); destination(d2); ...
    
            #another embedded log statement
            log {
                optional_element(filter1|parser1|rewrite1);
                optional_element(filter2|parser2|rewrite2);
                ...
                destination(d1); destination(d2); ...
            };
        };
        #set flags after the embedded log statements
        flags(flag1[, flag2...]);
    };

Example: Using embedded log paths

The following log path sends every message to the configured destinations: both the d_file1 and the d_file2 destinations receive every message of the source.

   log {
        source(s_localhost);
        destination(d_file1);
        destination(d_file2);
    };

The next example is equivalent to the one above, but uses an embedded log statement.

   log {
        source(s_localhost);
        destination(d_file1);
        log {
            destination(d_file2);
        };
    };

The following example uses two filters:

• messages coming from the host 192.168.1.1 are sent to the d_file1 destination, and

• messages coming from the host 192.168.1.1 and containing the string example are sent to the d_file2 destination.

   log {
        source(s_localhost);
        filter {
            host(192.168.1.1);
        };
        destination(d_file1);
        log {
            message("example");
            destination(d_file2);
        };
    };

The following example collects logs from multiple source groups and uses the source() filter in the embedded log statement to select messages of the s_network source group.

   log {
        source(s_localhost);
        source(s_network);
        destination(d_file1);
        log {
            filter {
                source(s_network);
            };
        destination(d_file2);
        };
    };

1.2 - if-else-elif: Conditional expressions

You can use if {}, elif {}, and else {} blocks to configure conditional expressions.

Conditional expressions’ format

Conditional expressions have two formats:

• Explicit filter expression:

      if (message('foo')) {
          parser { date-parser(); };
      } else {
          ...
      };
  

  This format only uses the filter expression in if(). If if does not contain 'foo', the else branch is taken.

  The else{} branch can be empty, you can use it to send the message to the default branch.

• Condition embedded in the log path:

      if {
          filter { message('foo')); };
          parser { date-parser(); };
      } else {
          ...
      };
  

  This format considers all filters and all parsers as the condition, combined. If the message contains 'foo' and the date-parser() fails, the else branch is taken. Similarly, if the message does not contain 'foo', the else branch is taken.

Using the if {} and else {} blocks in your configuration

You can copy-paste the following example and use it as a template for using the if {} and else {} blocks in your configuration.

Example for using the if {} and else {} blocks in your configuration

The following configuration can be used as a template for using the if {} and else {} blocks:

   log{
      source { example-msg-generator(num(1) template("...,STRING-TO-MATCH,..."));};
      source { example-msg-generator(num(1) template("...,NO-MATCH,..."));};
     
    if (message("STRING-TO-MATCH")) 
       {   
        destination { file(/dev/stdout template("matched: $MSG\n") persist-name("1")); };
       }
    else    
       {
        destination { file(/dev/stdout template("unmatched: $MSG\n") persist-name("2")); };
       };
    };

The configuration results in the following console printout:

   matched: ...,STRING-TO-MATCH,...
    unmatched: ...,NO-MATCH,...

An alternative, less straightforward way to implement conditional evaluation is to use junctions. For details on junctions and channels, see Junctions and channels.

1.3 - Junctions and channels

Junctions make it possible to send the messages to different channels, process the messages differently on each channel, and then join every channel together again. You can define any number of channels in a junction: every channel receives a copy of every message that reaches the junction. Every channel can process the messages differently, and at the end of the junction, the processed messages of every channel return to the junction again, where further processing is possible.

A junction includes one or more channels. A channel usually includes at least one filter, though that is not enforced. Otherwise, channels are identical to log statements, and can include any kind of objects, for example, parsers, rewrite rules, destinations, and so on. (For details on using channels, as well as on using channels outside junctions, see Using channels in configuration objects.)

You can also use log-path flags in the channels of the junction. Within the junction, a message is processed by every channel, in the order the channels appear in the configuration file. Typically if your channels have filters, you also set the flags(final) option for the channel. However, note that the log-path flags of the channel apply only within the junction, for example, if you set the final flag for a channel, then the subsequent channels of the junction will not receive the message, but this does not affect any other log path or junction of the configuration. The only exception is the flow-control flag: if you enable flow-control in a junction, it affects the entire log path. For details on log-path flags, see Log path flags.

   junction {
        channel { <other-syslog-ng-objects> <log-path-flags>};
        channel { <other-syslog-ng-objects> <log-path-flags>};
        ...
    };

Example: Using junctions

For example, suppose that you have a single network source that receives log messages from different devices, and some devices send messages that are not RFC-compliant (some routers are notorious for that). To solve this problem in earlier versions of AxoSyslog, you had to create two different network sources using different IP addresses or ports: one that received the RFC-compliant messages, and one that received the improperly formatted messages (for example, using the flags(no-parse) option). Using junctions this becomes much more simple: you can use a single network source to receive every message, then use a junction and two channels. The first channel processes the RFC-compliant messages, the second everything else. At the end, every message is stored in a single file. The filters used in the example can be host() filters (if you have a list of the IP addresses of the devices sending non-compliant messages), but that depends on your environment.

   log {
        source {
            syslog(
                ip(10.1.2.3)
                transport("tcp")
                flags(no-parse)
            );
        };
        junction {
            channel {
                filter(f_compliant_hosts);
                parser {
                    syslog-parser();
                };
            };
            channel {
                filter(f_noncompliant_hosts);
            };
        };
        destination {
            file("/var/log/messages");
        };
    };

Since every channel receives every message that reaches the junction, use the flags(final) option in the channels to avoid the unnecessary processing the messages multiple times:

   log {
        source {
            syslog(
                ip(10.1.2.3)
                transport("tcp")
                flags(no-parse)
            );
        };
        junction {
            channel {
                filter(f_compliant_hosts);
                parser {
                    syslog-parser();
                };
                flags(final);
            };
            channel {
                filter(f_noncompliant_hosts);
                flags(final);
            };
        };
        destination {
            file("/var/log/messages");
        };
    };

An alternative, more straightforward way to implement conditional evaluation is to configure conditional expressions using if {}, elif {}, and else {} blocks. For details, see if-else-elif: Conditional expressions.

1.4 - Log path flags

Flags influence the behavior of syslog-ng, and the way it processes messages. The following flags may be used in the log paths, as described in Log paths.

Log statement flags

catchall

This flag means that the source of the message is ignored, only the filters of the log path are taken into account when matching messages. A log statement using the catchall flag processes every message that arrives to any of the defined sources.

drop-unmatched

This flag means that the message is dropped along a log path when it does not match a filter or is discarded by a parser. Without using the drop-unmatched flag, AxoSyslog would continue to process the message along alternative paths.

fallback

This flag makes a log statement ‘fallback’. Fallback log statements process messages that were not processed by other, ’non-fallback’ log statements.

Processed means that every filter of a log path matched the message. Note that in the case of embedded log paths, the message is considered to be processed if it matches the filters of the outer log path, even if it does not match the filters of the embedded log path. For details, see Example: Using log path flags.

final

This flag means that the processing of log messages processed by the log statement ends here, other log statements appearing later in the configuration file will not process the messages processed by the log statement labeled as ‘final’. Note that this does not necessarily mean that matching messages will be stored only once, as there can be matching log statements processed before the current one (AxoSyslog evaluates log statements in the order they appear in the configuration file).

Processed means that every filter of a log path matched the message. Note that in the case of embedded log paths, the message is considered to be processed if it matches the filters of the outer log path, even if it does not match the filters of the embedded log path. For details, see Example: Using log path flags.

flow-control

Enables flow-control to the log path, meaning that AxoSyslog will stop reading messages from the sources of this log statement if the destinations are not able to process the messages at the required speed. If disabled, AxoSyslog will drop messages if the destination queues are full. If enabled, AxoSyslog will only drop messages if the destination queues/window sizes are improperly sized. For details, see Managing incoming and outgoing messages with flow-control.

Example: Using log path flags

Let’s suppose that you have two hosts (myhost_A and myhost_B) that run two applications each (application_A and application_B), and you collect the log messages to a central AxoSyslog server. On the server, you create two log paths:

• one that processes only the messages sent by myhost_A, and

• one that processes only the messages sent by application_A.

This means that messages sent by application_A running on myhost_A will be processed by both log paths, and the messages of application_B running on myhost_B will not be processed at all.

• If you add the final flag to the first log path, then only this log path will process the messages of myhost_A, so the second log path will receive only the messages of application_A running on myhost_B.

• If you create a third log path that includes the fallback flag, it will process the messages not processed by the first two log paths, in this case, the messages of application_B running on myhost_B.

• Adding a fourth log path with the catchall flag would process every message received by the AxoSyslog server.

      log { source(s_localhost); destination(d_file); flags(catchall); };
  

The following example shows a scenario that can result in message loss. Do NOT use such a configuration, unless you know exactly what you are doing. The problem is if a message matches the filters in the first part of the first log path, AxoSyslog treats the message as ‘processed’. Since the first log path includes the final flag, AxoSyslog will not pass the message to the second log path (the one with the fallback flag). As a result, AxoSyslog drops messages that do not match the filter of the embedded log path.

   # Do not use such a configuration, unless you know exactly what you are doing.
    log {
        source(s_network);
        # Filters in the external log path.
        # If a message matches this filter, it is treated as 'processed'
        filter(f_program);
        filter(f_message);
        log {
            # Filter in the embedded log path.
            # If a message does not match this filter, it is lost, it will not be processed by the 'fallback' log path
            filter(f_host);
            destination(d_file1);
        };
        flags(final);
    };
    
    log {
        source(s_network);
        destination(d_file2);
        flags(fallback);
    };

Example: Using the drop-unmatched flag

In the following example, if a log message arrives whose $MSG part does not contain the string foo, then AxoSyslog will discard the message and will not check compliance with the second if condition.

   ...
    if {
        filter { message('foo') };
        flags(drop-unmatched)
    };
    if {
        filter { message('bar') };
    };
    ...

(Without the drop-unmatched flag, AxoSyslog would check if the message complies with the second if condition, that is, whether or not the message contains the string bar .)

2 - Managing incoming and outgoing messages with flow-control

This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message losses.

The AxoSyslog application monitors (polls) the sources defined in its configuration file, periodically checking each source for messages. When a log message is found in one of the sources, AxoSyslog polls every source and reads the available messages. These messages are processed and put into the output buffer of AxoSyslog (also called fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.

In large-traffic environments many messages can arrive during a single poll loop, therefore AxoSyslog reads only a fixed number of messages from each source. The log-fetch-limit() option specifies the number of messages read during a poll loop from a single source.

TCP and unix-stream sources can receive the logs from several incoming connections (for example, many different clients or applications). For such sources, AxoSyslog reads messages from every connection, thus the log-fetch-limit() parameter applies individually to every connection of the source.

Log paths without flow-control

Every destination has its own output buffer. The output buffer is needed because the destination might not be able to accept all messages immediately. The log-fifo-size() parameter sets the size of the output buffer. The output buffer must be larger than the log-fetch-limit() of the sources, to ensure that every message read during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output buffer must be large enough to store the incoming messages of every source.

Log paths with flow-control

The AxoSyslog application uses flow-control in the following cases:

• Hard flow-control: the flow-control flag is enabled for the particular log path.

• Soft flow-control: the log path includes a file destination.

The flow-control of AxoSyslog introduces a control window to the source that tracks how many messages can AxoSyslog accept from the source. Every message that AxoSyslog reads from the source lowers the window size by one, every message that AxoSyslog successfully sends from the output buffer increases the window size by one. If the window is full (that is, its size decreases to zero), AxoSyslog stops reading messages from the source. The initial size of the control window is by default 100. If a source accepts messages from multiple connections, all messages use the same control window.

When using flow-control, AxoSyslog automatically sets the size of the output buffer so that it matches the size of the control window of the sources. Note that starting with AxoSyslog 3.22, log-fifo-size() only affects log paths that are not flow-controlled.

Dynamic flow-control

In addition to the static control window set using the log-iw-size() option, you can also allocate a dynamic window to the source. The AxoSyslog application uses this window to dynamically increase the static window of the active connections. The dynamic window is distributed evenly among the active connections of the source. The AxoSyslog application periodically checks which connections of the source are active, and redistributes the dynamic window. If only one of the connections is active, it receives the entire dynamic window, while other connections receive only their share of the static window.

Using dynamic flow-control on your AxoSyslog server is useful when the source has lots of connections, but only a small subset of the active clients send messages at high rate, and the memory of the AxoSyslog server is limited. In other cases, it is currently not recommended, because it can result in higher memory usage and fluctuating performance compared to using only the static window.

When flow-control is used, every source has its own control window. As a worst-case situation, memory of the host must be greater than the total size of the messages of every control window, plus the size of the dynamic window, that is, the log-iw-size()+dynamic-window-size(). This applies to every source that sends logs to the particular destination. Thus if two sources having several connections and heavy traffic send logs to the same destination, the control window of both sources must fit into the memory of the host. Otherwise, some messages might not fit in the memory, and messages may be lost.

If dynamic flow-control is disabled (which is the default behavior), the value of the log-iw-size() option cannot be lower than 100. If dynamic flow-control is enabled, you can decrease the value of the log-iw-size() option (to the minimum of 1).

In case of soft flow-control there is no message lost if the destination can accept messages. It is possible to lose messages if it cannot accept messages (for example, the file destination is not writable, or the disk becomes full), and all buffers are full. Soft flow-control cannot be configured, it is automatically available for file destinations.

Hard flow-control: In case of hard flow-control there is no message lost. To use hard flow-control, enable the flow-control flag in the log path. Hard flow-control is available for all destinations.

Example: Soft flow-control

   source s_file {
        file("/tmp/input_file.log");
    };
    destination d_file {
        file("/tmp/output_file.log");
    };
    destination d_tcp {
        network("127.0.0.1"
            port(2222)
        );
    };
    log {
        source(s_file);
        destination(d_file);
        destination(d_tcp);
    };

Example: Hard flow-control

   source s_file {
        file("/tmp/input_file.log");
    };
    destination d_file {
        file("/tmp/output_file.log");
    };
    destination d_tcp {
        network("127.0.0.1"
            port(2222)
        );
    };
    log {
        source(s_file);
        destination(d_file);
        destination(d_tcp);
        flags(flow-control);
    };

Handling outgoing messages

The AxoSyslog application handles outgoing messages the following way:

• Output queue: Messages from the output queue are sent to the target AxoSyslog server. The AxoSyslog application puts the outgoing messages directly into the output queue, unless the output queue is full. The output queue can hold 64 messages, this is a fixed value and cannot be modified.

• Disk buffer: If the output queue is full and disk-buffering is enabled, AxoSyslog puts the outgoing messages into the disk buffer of the destination.

• Overflow queue: If the output queue is full and the disk buffer is disabled or full, AxoSyslog puts the outgoing messages into the overflow queue of the destination. (The overflow queue is identical to the output buffer used by other destinations.) The log-fifo-size() parameter specifies the number of messages stored in the overflow queue, unless flow-control is enabled. When dynamic flow-control is enabled, AxoSyslog sets the size of the overflow queue automatically. For details on sizing the log-fifo-size() parameter, see Configuring flow-control.

2.1 - Flow-control and multiple destinations

Using flow-control on a source has an important side-effect if the messages of the source are sent to multiple destinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because AxoSyslog stops reading the source. For example, if messages from a source are sent to a remote server and also stored locally in a file, and the network connection to the server becomes unavailable, neither the remote server nor the local file will receive any messages.

If you use flow-control and reliable disk-based buffering together with multiple destinations, the flow-control starts slowing down the source only when:

• one destination is down, and

• the number of messages stored in the disk buffer of the destination reaches (capacity-bytes() minus flow-control-window-bytes()).

2.2 - Configuring flow-control

For details on how flow-control works, see Managing incoming and outgoing messages with flow-control. The summary of the main points is as follows:

• The AxoSyslog application normally reads a maximum of log-fetch-limit() number of messages from a source.

• From TCP and unix-stream sources, AxoSyslog reads a maximum of log-fetch-limit() from every connection of the source. The number of connections to the source is set using the max-connections() parameter.

• Every destination has an output buffer. The size of this buffer is set automatically for log paths that use flow-control, and can be set using the log-fifo-size() option for other log paths.

• Flow-control uses a control window to determine if there is free space in the output buffer for new messages. Every source has its own control window, the log-iw-size() option sets the size of the static control window. Optionally, you can enable a dynamic control window for the source using the dynamic-window-size() option.

• When a source accepts multiple connections, the size of the control window is divided by the value of the max-connections() parameter and this smaller control window is applied to each connection of the source.

  The dynamic control window is automatically distributed among the active connections of the source.

• If the control window is full, AxoSyslog stops reading messages from the source until some messages are successfully sent to the destination.

• If the output buffer becomes full, and neither disk-buffering nor flow-control is used, messages may be lost.

Example: Sizing parameters for flow-control

Suppose that AxoSyslog has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket.

Set the max-connections() parameter of the source to 300. However, the log-fetch-limit() (default value: 10) parameter applies to every connection of the source individually, while the log-iw-size() (default value: 1000) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log-fetch-limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max-connections()*log-fetch-limit() messages to be able to read every incoming message of a poll loop. In the current example this means that log-iw-size() should be greater than 300\*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing AxoSyslog to read only one message of the last connections in every poll loop.

The output buffer of the destination must accommodate at least log-iw-size() messages, but use a greater value: in the current example 3000\*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, AxoSyslog does not read any messages from the source until some messages are successfully sent to the destination.

   source s_localhost {
        network(
            ip(127.0.0.1)
            port(1999)
            max-connections(300)
        );
    };
    destination d_tcp {
        network("10.1.2.3"
            port(1999)
            localport(999)
            log-fifo-size(30000)
        );
    };
    log {
        source(s_localhost);
        destination(d_tcp);
        flags(flow-control);
    };

If other sources send messages to this destination, then the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, then increase the log-fifo-size() by 10000.

   source s_localhost {
        network(
            ip(127.0.0.1)
            port(1999)
            max-connections(300)
        );
    };
    source s_tcp {
        network(
            ip(192.168.1.5)
            port(1999)
            max-connections(100)
        );
    };
    destination d_tcp {
        network("10.1.2.3"
            port(1999)
            localport(999)
            log-fifo-size(40000)
        );
    };
    log {
        source(s_localhost);
        destination(d_tcp);
        flags(flow-control);
    };

3 - Using disk-based and memory buffering

The AxoSyslog application can store messages on the local hard disk if the destination (for example, the central log server) or the network connection to the destination becomes unavailable. The AxoSyslog application automatically sends the stored messages to the destination when the connection is reestablished. The disk buffer is used as a queue: when the connection to the destination is reestablished, AxoSyslog sends the messages to the destination in the order they were received.

Every destination driver supports the disk-buffer() option. The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method.

Every such destination uses a separate disk buffer (similarly to the output buffers controlled by log-fifo-size()). The hard disk space is not pre-allocated, so ensure that there is always enough free space to store the disk buffers even when the disk buffers are full.

If AxoSyslog is restarted (using the /etc/init.d/syslog-ng restart command, or another appropriate command on your platform), it automatically saves any unsent messages from the disk buffer and in-memory queues. After the restart, AxoSyslog sends the saved messages to the destination. In other words, the disk buffer is persistent. The disk buffer is also resistant to AxoSyslog crashes.

The AxoSyslog application supports two types of disk buffering: reliable and normal. For details, see Enabling reliable disk-based buffering and Enabling normal disk-based buffering, respectively.

Message handling and normal disk-based buffering

When you use disk-based buffering, and the reliable() option is set to no, AxoSyslog handles outgoing messages the following way:

• Output queue: In-memory queue. If there is space left in it, AxoSyslog puts the message into this queue first . Messages stored here are processed faster, because AxoSyslog can skip writing to, and reading from the disk, as well as serializing or deserializing the message, saving I/O and processor time as a result. The contents of the in-memory output queue are persisted to the disk-buffer file during AxoSyslog reload, restart or stop, but they cannot be persisted if in the event of power failures, or if AxoSyslog crashes. By default, the output queue can hold 1000 messages (you can adjust this number using the quot-size() option).

• Disk-buffer file: Disk queue. If there is no space left in the output queue, the message is stored on the disk-buffer file. Messages stored here are persisted on the disk, even in case of power failures or if AxoSyslog crashes. Using the disk-buffer file takes considerable amount of disk I/O and processor time. The size of this queue can be set with the capacity-bytes() option.

• Overflow queue: In-memory queue. This queue is used to trigger flow-control if it is set. The contents of the in-memory overflow queue are persisted to the disk-buffer file in case of AxoSyslog reload, restart or stop, but they are not persisted in case of power failures or if AxoSyslog crashes. Setting the size of the overflow queue can be done with the flow-control-window-size() option.

Message handling and reliable disk-based buffering

When you use disk-based buffering, and the reliable() option is set to yes, AxoSyslog handles outgoing messages the following way.

The flow-control-window-bytes() option determines when flow-control is triggered. After the size of the disk-buffer file reaches (capacity-bytes() minus flow-control-window-bytes()), messages are written into both the disk-buffer file and the overflow queue, indicating that flow-control needs to slow down the message source. These messages are not taken out from the control window (governed by log-iw-size()), causing the control window to fill up.

If the control window is full, the flow-control completely stops reading incoming messages from the source. (As a result, flow-control-window-bytes() must be at least as large as log-iw-size() times the average message size.)

• Output queue: In-memory and disk queue. If there is space left in it, AxoSyslog puts the message into this queue first. In case of reliable disk-buffer, in addition to storing the message in memory, it is stored directly in the disk-buffer file as well for safety reasons (see the next point). Messages stored here are processed faster, because AxoSyslog can skip reading from the disk, and deserializing the message, saving I/O and processor time. By default, the output queue can hold 1000 messages (you can adjust it using the quot-size() option).

• Disk-buffer file: Disk queue. If there is no space left in the output queue, the message is stored on the disk-buffer file. Messages stored here are persisted on the disk, and survive AxoSyslog crash or power failure. Using the disk-buffer file takes considerable amount of disk I/O and processor time. The size of this queue can be set with the capacity-bytes() option.

• Overflow queue: In-memory and disk queue. This queue is used to trigger flow-control if it is set. Similarly to the output queue, in case of reliable disk-buffer in addition to storing the message in memory, it is stored directly in the disk-buffer file as well for safety reasons. Setting the size of the overflow queue can be done with the flow-control-window-bytes() option.

3.1 - Enabling reliable disk-based buffering

Every destination driver supports the disk-buffer() option. The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method.

To enable reliable disk-based buffering, use the disk-buffer(reliable(yes)) parameter in the destination. Use reliable disk-based buffering if you do not want to lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. The filename of the reliable disk buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.rqf.

Example: Example for using reliable disk-based buffering

   destination d_BSD {
        network("127.0.0.1"
            port(3333)
            disk-buffer(
                flow-control-window-bytes(10000)
                capacity-bytes(2000000)
                reliable(yes)
            )
        );
    }; 

For details on the differences between normal and reliable disk-based buffering, see also About disk queue files.

3.2 - Enabling normal disk-based buffering

Every destination driver supports the disk-buffer() option. The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method.

If the reliable() option is not set, by default a normal disk-buffer is created. To explicitly enable the normal disk-buffer option, use the disk-buffer(reliable(no)) parameter in the destination. Use the normal disk-buffer option if you want a solution that is faster than the reliable disk-buffer option. In this case, the process will be less reliable and it is possible to lose logs in case of AxoSyslog crash. The filename of the normal disk-buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.qf.

Example: Example for using normal disk-based buffering

When using the disk-buffer plugin:

   destination d_BSD {
        network("127.0.0.1"
            port(3333)
            disk-buffer(
                flow-control-window-size(10000)
                capacity-bytes(2000000)
                reliable(no)
            )
        );
    }; 

For details on the differences between normal and reliable disk-based buffering, see also About disk queue files.

3.3 - How to get information about disk-buffer files

Purpose

This section describes how to get information about disk-buffer files used in AxoSyslog.

3.3.1 - Information about disk-buffer files

This section describes information about disk-buffer files used in AxoSyslog.

The following list contains information about how disk-buffer files are used in AxoSyslog:

• You can configure disk-buffer() for a remote destination in the destination() statement.

  For more information about an example of configuring disk-buffer() for a remote destination in the destination() statement, see disk-buffer().

• By default, AxoSyslog creates disk-buffer files under /opt/syslog-ng/var directory, unless dir() option is set in disk-buffer().

• The filenames are generated automatically by AxoSyslog with the extensions .qf for a normal disk-buffer and .rqf for a reliable disk-buffer.

• The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination, but doesn’t store information about the destination.

3.3.2 - Getting the status information of disk-buffer files

Purpose

This section describes getting the status information of the disk-buffer files used in AxoSyslog.

Command syntax

The basic command syntax for getting the status information of the disk-buffer files used in AxoSyslog looks like the following:

   /opt/syslog-ng/bin/dqtool info DISK-BUFFER_FILE

Example commands

The following example commands describe how you can get the status information of two different types of disk-buffer files (namely, empty normal disk-buffer files, and non-empty reliable disk-buffer queue files).

Example commands for empty, normal disk-buffer files, and non-empty, reliable disk-buffer queue files

• Empty, normal disk-buffer file

      /opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qfDisk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qf', number_of_messages='0'
  

• Non-empty, reliable disk-buffer queue file

      /opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.rqfReliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', number_of_messages='10'
  

One-liner command to get the state of disk-buffer files in the default directory

You can use the following one-liner command to get the state of disk-buffer files in the default directory:

   for qfile in /opt/syslog-ng/var/*.?(r)qf ; do /opt/syslog-ng/bin/dqtool info $qfile 2>&1 ; done

3.3.3 - Getting the list of disk-buffer files

Purpose

This section describes getting the list of disk-buffer files used in AxoSyslog.

The AxoSyslog application stores information (namely, the IP:PORT or DNS:PORT of the destinations, and the name of the disk-buffer file) about disk-buffer files in its persist file.

Example: command for listing the disk-buffer files in use

The following command will list the disk-buffer files in use:

   /opt/syslog-ng/bin/persist-tool dump /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist | awk -F '["=]' '/(qfile\(|\.queue)/ { gsub(/[ \t]+/, "", $5); gsub(/^[0-9A-Fa-f]{8}/, "", $5); "echo "$5"|xxd -r -p"|& getline QUEUE; printf("%s ==> %s\n",$1,QUEUE)}'

The example output will look like the following:

   afsocket_dd_qfile(stream,10.21.10.20:601)  ==> /opt/syslog-ng/var/syslog-ng-00000.rqf

   xxd: command not found

3.3.4 - Printing the content of disk-buffer files

Purpose

This section describes printing the content of the disk-buffer files used in AxoSyslog.

Command syntax

The command syntax for printing the content of the disk-buffer files used in AxoSyslog looks like the following:

   /opt/syslog-ng/bin/dqtool cat DISK-BUFFER_FILE

Short example output for printed content

Example: short output that shows the printed content of the disk-buffer files used in AxoSyslog

The following short output example shows the printed content of the disk-buffer files used in AxoSyslog:

   /opt/syslog-ng/bin/dqtool cat /opt/syslog-ng/var/syslog-ng-00000.rqf
    
    Reliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', queue_length='2952', size='-437712'
    Jul 31 12:33:48.226 10.21.10.10 <382019-07-31T12:33:36 localhost prg00000[1234]: seq: 0000000838, thread: 0000, runid: 1564569216, stamp: 2019-07-31T12:33:36 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
    ...

3.3.5 - Orphan disk-buffer files

Purpose

This section describes orphan disk-buffer files used in AxoSyslog.

Orphan disk-buffer files

In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), AxoSyslog creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.

Discovering the new disk-buffer files (orphan disk-buffer files)

To discover orphan disk-buffer files, get the list of disk-buffer files from the persist file, then compare the list with the contents of the disk-buffer files’ saving directory.

For more information about how you can get the list of disk-buffer files from the persist file, see Getting the list of disk-buffer files).

Example: difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files’ saving directory

The following examples show the difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files’ saving directory.

Disk-buffer file list from persist file:

   afsocket_dd_qfile(stream,10.21.10.112:514) = { "queue_file": "/opt/syslog-ng/var/syslog-ng-00001.rqf" }

Disk-buffer files’ saving directory content:

   # ls -l /opt/syslog-ng/var//var/lib/syslog-ng/*qf
    -rw------- 1 root root 2986780 Jul 31 12:30 /opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qf
    -rw------- 1 root root 2000080 Jul 31 12:31 /opt/syslog-ng/var/syslog-ng-00000.rqf
    -rw------- 1 root root    4096 Aug  1 11:09 /opt/syslog-ng/var/syslog-ng-00001.rqf

The disk-buffer files syslog-ng-00000.qf and syslog-ng-00000.rqf don’t exist in the persist file. These two files are the orphan disk-buffer files.

For more information about orphan disk-buffer files and how to process the messages in orphan disk-buffer files using a separate AxoSyslog instance, see How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance.

3.3.6 - How to process messages from an orphan disk-buffer file using a separate syslog-ng OSE instance

Purpose

This section describes how to read messages from an orphan disk-buffer file by using a separate AxoSyslog process running parallel to the already running AxoSyslog instance.

Orphan disk-buffer files

In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), AxoSyslog creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.

Processing the messages from an orphan disk-buffer file by using a separate AxoSyslog instance

When AxoSyslog creates orphan disk-buffer files, you can start a separate AxoSyslog instance parallel to the AxoSyslog instance already running, and use the following resolution process to process the messages in the orphan disk-buffer file.

To process the messages from an orphan disk-buffer file using a separate AxoSyslog instance,

1. Identify the orphan disk-buffer files and make a record of them. For more information, see How to get information about disk-buffer files.

   It is important to know the type of the disk-buffer file. Disk-buffer file types can be normal (.qf) or reliable (.rqf).

   In the examples during this process, the /opt/syslog-ng/var/syslog-ng-00005.rqf orphan reliable disk-buffer file is used.

2. Determine the destination of the logs. The content of the disk-buffer may help you determine the logs’ destination. For more information, see How to get information about disk-buffer files.

   In the examples during this process, the destination 10.21.10.20 is used with the standard network() port 514.

3. Create a directory for the temporary instance. In the examples during this process, the /tmp/qdisk directory is used.

       mkdir /tmp/qdisk
   

   Warning

   Make sure that there is sufficient disk space in the directory. The minimum recommended disk space in the directory is equal to the size of the orphan disk-buffer file.

4. Create the configuration file /tmp/qdisk/qdisk.conf for the temporary instance with the following content.

   Example: creating the /tmp/qdisk/qdisk.conf configuration file for the temporary instance

       @version:7.0
       @include "scl.conf"
   
       options {
         keep-hostname(yes);
         keep-timestamp(yes);
       };
   
       destination d_destination {
       #    ADD YOUR DESTINATION HERE
   
       };
   
       log {
         destination(d_destination);
       };
   

5. Add your destination statement with disk-buffer() to the configuration file. You can copy the destination statement from your running AxoSyslog configuration.

   Warning

   Add the dir() option and set the disk-buffer file’s destination directory to the temporary directory (that is, /tmp/qdisk) in your destination statement.

6. Start the temporary AxoSyslog instance in the foreground.

       syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl
   

   The AxoSyslog application will log to the console, so you will see any potential error that may occur during startup.

   The following example output displays that an empty disk-buffer file has been created and the connection to the remote destination has been established.

   Example: output displaying newly created empty disk-buffer file and connection established to remote destination

       Follow-mode file source not found, deferring open; filename='/no_such_file_or.dir'
       Reliable disk-buffer state saved; filename='/tmp/qdisk/syslog-ng-00000.rqf', qdisk_length='0'
       No server license found, running in client mode;
       syslog-ng starting up; version='7.0.20', cfg-fingerprint='eaa03b9efb88b87d7c1b0ce7efd042ed8ac0c013', cfg-nonce-ndx='0', cfg-signature='c0327a7f7e6418ce0399a75089377dfb662bb072'
       FIPS information; FIPS-mode='disabled'
       Syslog connection established; fd='7', server='AF_INET(10.21.10.20:514)', local='AF_INET(0.0.0.0:0)'
   

7. To stop AxoSyslog, press CTRL+C.

8. Overwrite the empty disk-buffer file with the orphan disk-buffer file.

       mv /opt/syslog-ng/var/syslog-ng-00005.rqf /tmp/qdisk/syslog-ng-00000.rqf
   

9. Start AxoSyslog using the command used in Start the temporary AxoSyslog instance in the foreground step.

       syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl
   

10. Open another terminal and check the progress by using one of the following methods.

    • Checking the number of stored logs in the disk-buffer (that is, the last number from the output).

          /opt/syslog-ng/sbin/syslog-ng-ctl stats -c /tmp/qdisk/qdisk.ctl | grep 'dst.*queued'
      

    • Checking the status of the disk-buffer file.

          /opt/syslog-ng/bin/dqtool info /tmp/qdisk/syslog-ng-00000.rqf
      

      An empty disk-buffer file will look similar to this:

      Example: empty disk-buffer file status message

      When checking the status of the disk-buffer files, the terminal will display a similar status message for an empty disk-buffer file:

          Reliable disk-buffer state loaded; filename='/tmp/qdisk/syslog-ng-00000.rqf', queue_length='0', size='0'
      

11. Press CTRL+C to stop AxoSyslog.

12. Check the state of the orphan disk-buffer file. For more information, see How to get information about disk-buffer files.

13. If you have more than one orphan disk-buffer file, repeat the steps following the AxoSyslog stop (that is, the steps beginning from overwriting the empty disk-buffer file with the orphan disk-buffer file) for each orphan disk-buffer file.

14. Remove the temporary directory.

    Example: command for removing the temporary directory

    The following command removes the /mp/qdisk temporary directory:

        rm -rf /tmp/qdisk
    

3.4 - Enabling memory buffering

To enable memory buffering, use the log-fifo-size() parameter in the destination. All destination drivers can use memory buffering. Use memory buffering if you want to send logs to destinations where disk-based buffering is not available. Or if you want the fastest solution, and if AxoSyslog crash or network downtime is never expected. In these cases, losing logs is possible. This solution does not use disk-based buffering, logs are stored only in the memory.

Example: Example for using memory buffering

   destination d_BSD {
        network("127.0.0.1"
            port(3333)
            log-fifo-size(10000)
        );
    };

3.5 - About disk queue files

Normal and reliable queue files

The key difference between disk queue files that employ the reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to AxoSyslog crashes. Of course, using the reliable(yes) option introduces a significant performance penalty as well.

Both reliable and normal disk-buffers employ an in-memory output queue (set in quot-size()) and an in-memory overflow queue (set in flow-control-window-bytes() for reliable disk-buffers, or flow-control-window-size() for normal disk-buffers). The difference between reliable and normal disk-buffers is that when the reliable disk-buffer uses one of its in-memory queues, it also stores the message on the disk, whereas the normal disk-buffer stores the message only in memory. The normal disk-buffer only uses the disk if the in-memory output buffer is filled up completely. This approach has better performance (due to fewer disk I/O operations), but also carries the risk of losing a maximum of quot-size() plus flow-control-window-size() number of messages in case of an unexpected power failure or application crash.

Size of the queue files

Disk queue files grow. Each may take up to capacity-bytes() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file.

The disk-buffer file’s size should be considered as the configured capacity-bytes() at any point of time, even if it does not have messages in it. Truncating the disk-buffer file can slow down disk I/O operations, so AxoSyslog does not always truncate the file when it would be possible (see the truncate-size-ratio() option). If a large disk-buffer file is not desirable, you should set the capacity-bytes() option to a smaller value. Note that AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default.

Starting with AxoSyslog version 4.0, you can preallocate disk-buffer files.

Preallocating disk-buffer files

By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).

In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.

If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).

4 - Filters

The following sections describe how to select and filter log messages.

• Using filters describes how to configure and use filters.

• Combining filters with boolean operators shows how to create complex filters using boolean operators.

• Comparing macro values in filters explains how to evaluate macros in filters.

• Using wildcards, special characters, and regular expressions in filters provides tips on using regular expressions.

• Tagging messages explains how to tag messages and how to filter on the tags.

• Filter functions is a detailed description of the filter functions available in AxoSyslog.

4.1 - Using filters

Filters perform log routing: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.

To define a filter, add a filter statement to the syslog-ng.conf configuration file using the following syntax:

   filter <identifier> { <filter_type>("<filter_expression>"); };

Then use the filter in a log path, for example:

   log {
        source(s1);
        filter(<identifier>);
        destination(d1); };

You can also define the filter inline. For details, see Defining configuration objects inline.

Example: A simple filter statement

The following filter statement selects the messages that contain the word deny and come from the host example.

   filter demo_filter { host("example") and match("deny" value("MESSAGE"))
    };
    log {
        source(s1);
        filter(demo_filter);
        destination(d1);
    };

The following example does the same, but defines the filter inline.

   log {
        source(s1);
        filter { host("example") and match("deny" value("MESSAGE")) };
        destination(d1);
    };

4.2 - Combining filters with boolean operators

When a log statement includes multiple filter statements, AxoSyslog sends a message to the destination only if all filters are true for the message. In other words, the filters are connected with the logical AND operator. In the following example, no message arrives to the destination, because the filters are exclusive (the hostname of a client cannot be example1 and example2 at the same time):

   filter demo_filter1 { host("example1"); };
    filter demo_filter2 { host("example2"); };
    log {
        source(s1); source(s2);
        filter(demo_filter1); filter(demo_filter2);
        destination(d1); destination(d2); };

To select the messages that come from either host example1 or example2, use a single filter expression:

   filter demo_filter { host("example1") or host("example2"); };
    log {
        source(s1); source(s2);
        filter(demo_filter);
        destination(d1); destination(d2); };

Use the not operator to invert filters, for example, to select the messages that were not sent by host example1:

   filter demo_filter { not host("example1"); };

However, to select the messages that were not sent by host example1 or example2, you have to use the and operator (that’s how boolean logic works):

   filter demo_filter { not host("example1") and not host("example2"); };

Alternatively, you can use parentheses to avoid this confusion:

   filter demo_filter { not (host("example1") or host("example2")); };

For a complete description on filter functions, see Filter functions.

The following filter statement selects the messages that contain the word deny and come from the host example.

   filter demo_filter { host("example") and match("deny" value("MESSAGE")); };

The value() parameter of the match function limits the scope of the function to the text part of the message (that is, the part returned by the ${MESSAGE} macro), or optionally to the content of any other macro. The template() parameter of the match function can be used to run a filter against a combination of macros. For details on using the match() filter function, see match().

4.3 - Comparing macro values in filters

In AxoSyslog you can compare macro values and templates as numerical and string values. String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string. For details on macros and templates, see Customize message format using macros and templates.

Use the following syntax to compare macro values or templates.

    filter <filter-id>
        {"<macro-or-template>" operator "<value-or-macro-or-template>"};

String and numerical comparison

AxoSyslog versions earlier than 4.0 used separate operators for string comparisons (for example, eq). In version 4.0 and later, you can simply use the mathematical symbols as operators (like ==, !=, >=), and AxoSyslog automatically determines how to compare the arguments from their type. The logic behind that is similar to JavaScript:

• If both sides of the comparisons are strings, then the comparison is string.
• If one of the arguments is numeric, then the comparison is numeric.
• Literal numbers (numbers not enclosed in quotes) are numeric.
• You can explicitly type-cast an argument into a number.

For example:

• if ("${.apache.httpversion}" == 1.0)

  The right side of the == operator is 1.0, which is a floating point literal, so the comparison is numeric.

• if (double("${.apache.httpversion}") == "1.0")

  The left side is explicitly type cast into double, the right side is string (because of the quotes), so the comparison is numeric.

• if ("${.apache.request}" == "/wp-admin/login.php")

  The left side is not type-cast, so it’s a string, the right side is a string, so the comparison is string.

Note: You can still use the old string operators if you want to, they are available for backwards compatibility.

Example: Comparing macro values in filters

The following expression selects log messages containing a PID (that is, ${PID} macro is not empty):

    filter f_pid {"${PID}" != ""};

The following expression selects log messages that do not contain a PID. Also, it uses a template as the left argument of the operator and compares the values as strings:

    filter f_pid {"${HOST}${PID}" == "${HOST}"};

The following example selects messages with priority level higher than 5.

    filter f_level {"${LEVEL_NUM}" > 5};

Make sure to:

• Enclose macros and templates in double-quotes.
• Use the $ character before macros.

Note that:

• You can use type casting anywhere where you can use templates to apply a type to the result of the template expansion.
• Using comparator operators can be equivalent to using filter functions, but is somewhat slower. For example, using "${HOST}" == "myhost" is equivalent to using host("myhost" type(string)).
• You can use any macro in the expression, including user-defined macros from parsers and results of pattern database classifications.
• The results of filter functions are boolean values, so they cannot be compared to other values.
• You can use boolean operators to combine comparison expressions.

Comparison operators

The following numerical and string comparison operators are available.

4.4 - Using wildcards, special characters, and regular expressions in filters

The host(), match(), and program() filter functions accept regular expressions as parameters. The exact type of the regular expression to use can be specified with the type() option. By default, AxoSyslog uses PCRE regular expressions.

In regular expressions, the asterisk (\) character means 0, 1, or any number of the previous expression. For example, in the f\*ilter expression the asterisk means 0 or more f letters. This expression matches for the following strings: ilter, filter, ffilter, and so on. To achieve the wildcard functionality commonly represented by the asterisk character in other applications, use .\ in your expressions, for example, f.\*ilter.

Alternatively, if you do not need regular expressions, only wildcards, use type(glob) in your filter:

Example: Filtering with widcards

The following filter matches on hostnames starting with the myhost string, for example, on myhost-1, myhost-2, and so on.

   filter f_wildcard {host("myhost*" type(glob));};

For details on using regular expressions in AxoSyslog, see Using wildcards, special characters, and regular expressions in filters.

To filter for special control characters like the carriage return (CR), use the \\r escape prefix in AxoSyslog version 3.0 and 3.1. In AxoSyslog 3.2 and later, you can also use the \\x escape prefix and the ASCII code of the character. For example, to filter on carriage returns, use the following filter:

   filter f_carriage_return {match("\x0d" value ("MESSAGE"));};

4.5 - Tagging messages

You can label the messages with custom tags. Tags are simple labels, identified by their names, which must be unique. Currently AxoSyslog can tag a message at the following places:

• At the source when the message is received.
  • AxoSyslog automatically adds the .source.<id_of_the_source_statement> tag to every incoming message.
  • You can add custom tag using the tags() option of the source.

  • AxoSyslog 4.7 and newer automatically adds the following tags if it encounters errors when parsing syslog messages.

    • message.utf8_sanitized
    • message.parse_error
    • syslog.missing_pri
    • syslog.missing_timestamp
    • syslog.invalid_hostname
    • syslog.unexpected_framing
    • syslog.rfc3164_missing_header
    • syslog.rfc5424_unquoted_sdata_value
• When the message matches a pattern in the pattern database. For details on using the pattern database, see Using pattern databases, for details on creating tags in the pattern database, see The pattern database format.
• Tags can be also added and deleted using rewrite rules. For details, see Adding and deleting tags.

You can use the tags() filter to select only specific messages.

• Tagging messages and also filtering on the tags is very fast, much faster than other types of filters.
• Tags are available locally, that is, if you add tags to a message on the client, these tags will not be available on the server.
• To include the tags in the message, use the ${TAGS} macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the ${TAGS} macro in the .SDATA.meta part of the message. Note that the ${TAGS} macro is available only in AxoSyslog 3.1.1 and later.

For an example on tagging, see Example: Adding tags and filtering messages with tags.

4.6 - Filter functions

The following functions may be used in the filter statement, as described in Filters.

Filter functions available in AxoSyslog

4.6.1 - facility()

Description: Match messages having one of the listed facility codes.

The facility() filter accepts both the name and the numerical code of the facility or the importance level. Facility codes 0-23 are predefined and can be referenced by their usual name. Facility codes above 24 are not defined.

You can use the facility filter the following ways:

• Use a single facility name, for example, facility(user)

• Use a single facility code, for example, facility(1)

• Use a facility range (works only with facility names), for example, facility(local0..local5)

The AxoSyslog application recognizes the following facilities: (Note that some of these facilities are available only on specific platforms.)

syslog Message Facilities recognized by the facility() filter

4.6.2 - filter()

Description: Call another filter rule and evaluate its value. For example:

   filter demo_filter { host("example") and match("deny" value("MESSAGE")) };
    filter inverted_demo_filter { not filter(demo_filter) }

4.6.3 - host()

Description: Match messages by using a regular expression against the hostname field of log messages. Note that you can filter only on the actual content of the HOST field of the message (or what it was rewritten to). That is, AxoSyslog will compare the filter expression to the content of the ${HOST} macro. This means that for the IP address of a host will not match, even if the IP address and the hostname field refers to the same host. To filter on IP addresses, use the netmask() filter.

   filter demo_filter { host("example") };

4.6.4 - in-list()

Description: Matches the value of the specified field to a list stored in a file, allowing you to do simple, file-based black- and whitelisting. The file must be a plain-text file, containing one entry per line.

The AxoSyslog application loads the entire file, and compares the value of the specified field (for example, ${PROGRAM}) to entries in the file. When you use the in-list() filter, note the following points:

• Comparing the values is case-sensitive.
• Only exact matches are supported, partial and substring matches are not.
• If you modify the list file, reload the configuration of AxoSyslog for the changes to take effect.
• Make sure that there is an empty line in the file after the last entry. If the last entry is the last line, it will be excluded from the list.

Available in AxoSyslog 3.5 and later.

Example: Selecting messages using the in-list() filter

Create a text file that contains the programs (as in the ${PROGRAM} field of their log messages) you want to select. For example, you want to forward only the logs of a few applications from a host: kernel, sshd, and sudo. Create the /etc/syslog-ng/programlist.list file with the following contents. Note the empty line after the last entry.

kernel
sshd
sudo

The following filter selects only the messages of the listed applications:

   filter f_whitelist { in-list("/etc/syslog-ng/programlist.list", value("PROGRAM")); };

Create the appropriate sources and destinations for your environment, then create a log path that uses the previous filter to select only the log messages of the applications you need:

   log {
        source(s_all);
        filter(f_whitelist);
        destination(d_logserver); };

To create a blacklist filter, simply negate the in-list() filter:

   filter f_blacklist { not in-list("/etc/syslog-ng/programlist.list", value("PROGRAM")); };

4.6.5 - level() or priority()

Description: The level() filter selects messages corresponding to a single importance level, or a level-range. To select messages of a specific level, use the name of the level as a filter parameter, for example, use the following to select warning messages:

   level(warning)

To select a range of levels, include the beginning and the ending level in the filter, separated with two dots (..). For example, to select every message of error or higher level, use the following filter:

   level(err..emerg)

The level() filter accepts the following levels: emerg, alert, crit, err, warning, notice, info, debug.

4.6.6 - match()

Description: Match a regular expression to the headers and the message itself (that is, the values returned by the MSGHDR and MSG macros). Note that in AxoSyslog version 2.1 and earlier, the match() filter was applied only to the text of the message, excluding the headers. This functionality has been moved to the message() filter.

To limit the scope of the match to a specific part of the message (identified with a macro), use the match(regexp value("MACRO")) syntax. Do not include the $ sign in the parameter of the value() option.

The value() parameter accepts both built-in macros and user-defined ones created with a parser or using a pattern database. For details on macros and parsers, see Templates and macros, Parsing messages with comma-separated and similar values, and Using parser results in filters and templates.

Starting with version 3.22, the match() filter can work on templates as well. This means that you can a match against an expression combined of macros, instead of a single macro. Note that when using a template, you must reference macros with the $ sign (unlike when using the value() parameter). For example:

   match("^my-regular-expression" template("${HOST}|${PROGRAM}${PID}|${MESSAGE}"));

Using a template with a single macro is equivalent with using the value() parameter. For example, the following two lines are equivalent:

   match("^my-regular-expression" value("MESSAGE"));
    match("^my-regular-expression" template("${MESSAGE}"));

4.6.7 - message()

Description: Match a regular expression to the text of the log message, excluding the headers (that is, the value returned by the MSG macros). Note that in syslog-ng version 2.1 and earlier, this functionality was performed by the match() filter.

4.6.8 - netmask()

Description: Select only messages sent by a host whose IP address belongs to the specified IPv4 subnet. Note that this filter checks the IP address of the last-hop relay (the host that actually sent the message to AxoSyslog), not the contents of the HOST field of the message. You can use both the dot-decimal and the CIDR notation to specify the netmask. For example, 192.168.5.0/255.255.255.0 or 192.168.5.0/24. To filter IPv6 addresses, see netmask6().

4.6.9 - netmask6()

Description: Select only messages sent by a host whose IP address belongs to the specified IPv6 subnet. Note that this filter checks the IP address of the last-hop relay (the host that actually sent the message to AxoSyslog), not the contents of the HOST field of the message. You can use both the regular and the compressed format to specify the IP address, for example, 1080:0:0:0:8:800:200C:417A or 1080::8:800:200C:417A. If you do not specify the address, localhost is used.

Use the netmask (also called prefix) to specify how many of the leftmost bits of the address comprise the netmask (values 1-128 are valid). For example, the following specify a 60-bit prefix: 12AB:0000:0000:CD30:0000:0000:0000:0000/60 or 12AB::CD30:0:0:0:0/60. Note that if you set an IP address and a prefix, AxoSyslog will ignore the bits of the address after the prefix. To filter IPv4 addresses, see netmask().

The netmask6() filter is available in AxoSyslog 3.7 and later.

4.6.10 - program()

Description: Match messages by using a regular expression against the program name field of log messages.

4.6.11 - rate-limit()

Description: Limits messages rate based on arbitrary keys in each message. The key will be resolved using the key() option. Each resolution will be allowed to have the number of messages each second, set by the rate() option. For example if key($HOST) and rate(5000) are set, and there are 2 hosts sending messages to AxoSyslog, a total of 10000 messages will be allowed by the rate-limit() filter, 5000 from the first and 5000 from the second host. If key() was not set instead, then 5000 messages would be allowed each second, regardless of their content.

Example: Using the rate-limit() filter

The following example depicts the scenario described in the description part of this section.

   filter f_rate_limit {
                    rate-limit(
                    key("$HOST")
                    rate(5000)
                    );
                };

4.6.11.1 - Options of rate-limit() filter

The rate-limit() filter has the following options.

key()

Description: The resolved template, that will be used to create unique rate-limit token buckets. In AxoSyslog version 4.4 and earlier, the name of this option was template().

rate()

Description: The number of messages for each unique macro resolution, that will be let through (matched) by the filter each second.

4.6.12 - source()

Description: Select messages of a source statement. This filter can be used in embedded log statements if the parent statement contains multiple source groups — only messages originating from the selected source group are sent to the destination of the embedded log statement.

4.6.13 - tags()

Description: Select messages labeled with the specified tag. Every message automatically has the tag of its source in .source.<id_of_the_source_statement> format. This option is available only in AxoSyslog 3.1 and later.

Example: Adding tags and filtering messages with tags

   source s_tcp {
        network(ip(192.168.1.1) port(1514) tags("tcp", "router"));
    };

Use the tags() option of the filters to select only specific messages:

   filter f_tcp {
        tags(".source.s_tcp");
    };
    
    filter f_router {
        tags("router");
    };

   filter f_tag_filter {tags(".classifier.system");};

5 - Dropping messages

To skip the processing of a message without sending it to a destination, create a log statement with the appropriate filters, but do not include any destination in the statement, and use the final flag.

Example: Skipping messages

The following log statement drops all debug level messages without any further processing.

   filter demo_debugfilter { level(debug); };
    log { source(s_all); filter(demo_debugfilter); flags(final); };