parser: Parse and segment structured messages

• 1: Apache access log parser
• 1.1: Options of apache-accesslog-parser() parsers
• 2: Check Point Log Exporter parser
• 3: Cisco parser
• 4: Parsing messages with comma-separated and similar values
• 4.1: Options of CSV parsers
• 5: Parsing dates and timestamps
• 5.1: Options of date-parser() parsers
• 6: db-parser: Process message content with a pattern database (patterndb)
• 6.1: Classifying log messages
• 6.1.1: The structure of the pattern database
• 6.1.2: How pattern matching works
• 6.1.3: Artificial ignorance
• 6.2: Using pattern databases
• 6.2.1: Using parser results in filters and templates
• 6.2.2: Downloading sample pattern databases
• 6.2.3: Correlating log messages using pattern databases
• 6.2.3.1: Referencing earlier messages of the context
• 6.3: Triggering actions for identified messages
• 6.3.1: Conditional actions
• 6.3.2: External actions
• 6.3.3: Actions and message correlation
• 6.4: Creating pattern databases
• 6.4.1: Using pattern parsers
• 6.4.1.1: Pattern parsers
• 6.4.2: What's new in the pattern database format V5
• 6.4.3: The pattern database format
• 6.4.3.1: Element: patterndb
• 6.4.3.2: Element: ruleset
• 6.4.3.3: Element: patterns
• 6.4.3.4: Element: rules
• 6.4.3.5: Element: rule
• 6.4.3.6: Element: patterns
• 6.4.3.7: Element: urls
• 6.4.3.8: Element: values
• 6.4.3.9: Element: actions
• 6.4.3.10: Element: action
• 6.4.3.11: Element: create-context
• 6.4.3.12: Element: tags
• 6.4.3.13: Element: example
• 6.4.3.14: Element: examples
• 7: Parsing enterprise-wide message model (EWMM) messages
• 8: Fortigate parser
• 8.1: Fortigate parser options
• 9: group-lines parser
• 10: iptables parser
• 11: JSON parser
• 11.1: Options of JSON parsers
• 12: Parsing key=value pairs
• 12.1: Options of key=value parsers
• 13: Linux audit parser
• 13.1: Options of linux-audit-parser() parsers
• 14: MariaDB parser
• 15: metrics-probe
• 16: Netskope parser
• 17: Parse OpenTelemetry messages
• 18: panos-parser(): parsing PAN-OS log messages
• 18.1: Message format parsed by panos-parser()
• 18.2: PAN-OS parser options
• 19: PostgreSQL csvlog
• 20: Python parser
• 21: Regular expression (regexp) parser
• 21.1: Options of Regular expression parsers
• 22: Structured data (SDATA) parser
• 23: Sudo parser
• 24: Parsing syslog messages
• 24.1: Options of syslog-parser() parsers
• 25: Parsing tags
• 26: Websense parser
• 27: Windows XML Event Log (EVTX) parser
• 28: XML parser
• 28.1: Limitations of the XML parsers
• 28.2: Options of the XML parsers

1 - Apache access log parser

The Apache access log parser can parse the access log messages of the Apache HTTP Server. The AxoSyslog application can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The apache-accesslog-parser() supports both the Common Log Format and the Combined Log Format of Apache (for details, see the Apache HTTP Server documentation). The following is a sample log message:

   127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

Starting with version 3.21, virtualhost and the port of the virtualhost (vhost) is also supported, for example:

   foo.com:443 1.2.3.4 - - [15/Apr/2019:14:30:16 -0400] "GET /bar.html HTTP/2.0" 500 - "https://foo.com/referer.html" "Mozilla/5.0 ..."

The AxoSyslog application extracts every field into name-value pairs, and adds the .apache. prefix to the name of the field.

Declaration:

   parser parser_name {
        apache-accesslog-parser(
            prefix()
        );
    };

The parser extracts the following fields from the messages: vhost, port, clientip, ident, auth, timestamp, rawrequest, response, bytes, referrer, and agent. The rawrequest field is further segmented into the verb, request, and httpversion fields. The AxoSyslog apache-accesslog-parser() parser uses the same naming convention as Logstash.

Example: Using the apache-accesslog-parser parser

In the following example, the source is a log file created by an Apache web server. The parser automatically inserts the .apache. prefix before all extracted name-value pairs. The destination is a file that uses the format-json template function. Every name-value pair that begins with a dot (.) character will be written to the file (dot-nv-pairs). The log statement connects the source, the destination, and the parser.

   source s_apache {
        file(/var/log/access_log);
    };
    
    destination d_json {
        file(
            "/tmp/test.json"
            template("$(format-json .apache.*)\n")
        );
    };
    
    log {
        source(s_apache);
        parser { apache-accesslog-parser();};
        destination(d_json);
    };

To use this parser, the scl.conf file must be included in your AxoSyslog configuration:

   @include "scl.conf"

The apache-accesslog-parser() is actually a reusable configuration snippet configured parse Apache access log messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

1.1 - Options of apache-accesslog-parser() parsers

The apache-accesslog-parser() has the following options.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, apache-accesslog-parser() uses the .apache. prefix. To modify it, use the following format:

   parser {
        apache-accesslog-parser(prefix("apache."));
    };

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

2 - Check Point Log Exporter parser

The Check Point Log Exporter parser can parse Check Point log messages. These messages do not completely comply with the syslog RFCs, making them difficult to parse. The checkpoint-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following formats:

   <PRI><VERSION> <YYYY-MM-DD> <HH-MM-SS> <PROGRAM> <PID> <MSGID> - [key1:value1; key2:value2; ... ]

For example:

   <134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}";]

Splunk format:

   time=1557767758|hostname=r80test|product=Firewall|layer_name=Network|layer_uuid=c0264a80-1832-4fce-8a90-d0849dc4ba33|match_id=1|parent_rule=0|rule_action=Accept|rule_uid=4420bdc0-19f3-4a3e-8954-03b742cd3aee|action=Accept|ifdir=inbound|ifname=eth0|logid=0|loguid={0x5cd9a64e,0x0,0x5060a8c0,0xc0000001}|origin=192.168.96.80|originsicname=cn\=cp_mgmt,o\=r80test..ymydp2|sequencenum=1|time=1557767758|version=5|dst=192.168.96.80|inzone=Internal|outzone=Local|proto=6|s_port=63945|service=443|service_id=https|src=192.168.96.27|

If you find a message that the checkpoint-parser() cannot properly parse, contact us, so we can improve the parser.

By default, the Check Point-specific fields are extracted into name-value pairs prefixed with .checkpoint. For example, the action in the previous message becomes ${.checkpoint.action}. You can change the prefix using the prefix option of the parser.

Declaration:

   @version: 4.5.0
    @include "scl.conf"
    log {
        source { network(flags(no-parse)); };
        parser { checkpoint-parser(); };
        destination { ... };
    };

Note that the parser expects that the entire incorrectly formatted syslog message (starting with its <PRI> value) is in $MSG, which you can achieve by using flags(no-parse) on the input driver.

The checkpoint-parser() is actually a reusable configuration snippet configured to parse Check Point messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, checkpoint-parser() uses the .checkpoint. prefix. To modify it, use the following format:

   parser {
        checkpoint-parser(prefix("myprefix."));
    };

3 - Cisco parser

The Cisco parser can parse the log messages of various Cisco devices. The messages of these devices often do not completely comply with the syslog RFCs, making them difficult to parse. The cisco-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs, extracting also the Cisco-specific values, for example, the mnemonic. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse variations of the following message format:

   <pri>(sequence: )?(origin-id: )?(timestamp? timezone?: )?%msg

For example:

   <189>29: foo: *Apr 29 13:58:40.411: %SYS-5-CONFIG_I: Configured from console by console
    <190>30: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 stopped - CLI initiated
    <190>31: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 started - CLI initiated
    <189>32: 0.0.0.0: *Apr 29 13:59:12.491: %SYS-5-CONFIG_I: Configured from console by console
    <189>32: foo: *Apr 29 13:58:46.411: %SYSMGR-STANDBY-3-SHUTDOWN_START: The System Manager has started the shutdown procedure.

The AxoSyslog application normalizes the parsed log messages into the following format:

   ${MESSAGE}=%FAC-SEV-MNEMONIC: message
    ${HOST}=origin-id

By default, the Cisco-specific fields are extracted into the following name-value pairs:${.cisco.facility}, ${.cisco.severity}, ${.cisco.mnemonic}. You can change the prefix using the prefix option.

Declaration:

@version: 4.5.0
@include "scl.conf"
log {
    source { network(
                transport("udp")
                flags(no-parse)
                ); };
    parser { cisco-parser(); };
    destination { ... };
};

Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.

The cisco-parser() is actually a reusable configuration snippet configured to parse Cisco messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, cisco-parser() uses the .cisco. prefix. To modify it, use the following format:

   parser {
        cisco-parser(prefix("myprefix."));
    };

4 - Parsing messages with comma-separated and similar values

The AxoSyslog application can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) at delimiter characters or strings to named fields (columns) using the csv (comma-separated-values) parser The parsed fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.

Parsers are similar to filters: they must be defined in the AxoSyslog configuration file and used in the log statement. You can also define the parser inline in the log path.

To create a csv-parser(), you have to define the columns of the message, the separator characters or strings (also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used to escape the delimiter characters (quote-pairs()).

Declaration:

   parser <parser_name> {
        csv-parser(
            columns(column1, column2, ...)
            delimiters(chars("<delimiter_characters>"), strings("<delimiter_strings>"))
        );
    };

Column names work like macros.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

Starting with AxoSyslog version 4.5, you can omit the columns() option, and extract the values into matches ($1, $2, $3, and so on), which are available as the anonymous list $*. For example:

@version: current

log {
    source { tcp(port(2000) flags(no-parse)); };

    parser { csv-parser(delimiters(',') dialect(escape-backslash)); };
    destination { stdout(template("$ISODATE $*\n")); };
};

Example: Segmenting hostnames separated with a dash

The following example separates hostnames like example-1 and example-2 into two parts.

   parser p_hostname_segmentation {
        csv-parser(columns("HOSTNAME.NAME", "HOSTNAME.ID")
        delimiters("-")
        flags(escape-none)
        template("${HOST}"));
    };
    destination d_file {
        file("/var/log/messages-${HOSTNAME.NAME:-examplehost}");
    };
    log {
        source(s_local);
        parser(p_hostname_segmentation);
        destination(d_file);
    };

Example: Parsing Apache log files

The following parser processes the log of Apache web servers and separates them into different fields. Apache log messages can be formatted like:

   "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %v"

Here is a sample message:

   192.168.1.1 - - [31/Dec/2007:00:17:10 +0100] "GET /cgi-bin/example.cgi HTTP/1.1" 200 2708 "-" "curl/7.15.5 (i4 86-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5" 2 example.mycompany

To parse such logs, the delimiter character is set to a single whitespace (delimiters(" ")). Whitespaces between quotes and brackets are ignored (quote-pairs('""[]')).

   parser p_apache {
        csv-parser(
            columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",
            "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",
            "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",
            "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")
            flags(escape-double-char,strip-whitespace)
            delimiters(" ")
            quote-pairs('""[]')
        );
    };

The results can be used for example, to separate log messages into different files based on the APACHE.USER_NAME field. If the field is empty, the nouser name is assigned.

   log {
        source(s_local);
        parser(p_apache);
        destination(d_file);
    };
    destination d_file {
        file("/var/log/messages-${APACHE.USER_NAME:-nouser}");
    };

Example: Segmenting a part of a message

Multiple parsers can be used to split a part of an already parsed message into further segments. The following example splits the timestamp of a parsed Apache log message into separate fields.

   parser p_apache_timestamp {
        csv-parser(
            columns("APACHE.TIMESTAMP.DAY", "APACHE.TIMESTAMP.MONTH", "APACHE.TIMESTAMP.YEAR", "APACHE.TIMESTAMP.HOUR", "APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.SEC", "APACHE.TIMESTAMP.ZONE")
            delimiters("/: ")
            flags(escape-none)
            template("${APACHE.TIMESTAMP}")
        );
    };
    log {
        source(s_local);
        parser(p_apache);
        parser(p_apache_timestamp);
        destination(d_file);
    };

Further examples:

• For an example on using the greedy option, see Example: Adding the end of the message to the last column.

4.1 - Options of CSV parsers

The AxoSyslog application can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) at delimiter characters or strings to named fields (columns) using the csv (comma-separated-values) parser The parsed fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.

Parsers are similar to filters: they must be defined in the AxoSyslog configuration file and used in the log statement. You can also define the parser inline in the log path.

To create a csv-parser(), you have to define the columns of the message, the separator characters or strings (also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used to escape the delimiter characters (quote-pairs()).

Declaration:

   parser <parser_name> {
        csv-parser(
            columns(column1, column2, ...)
            delimiters(chars("<delimiter_characters>"), strings("<delimiter_strings>"))
        );
    };

Column names work like macros.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

columns()

Description: Specifies the name of the columns to separate messages to. These names will be automatically available as macros. The values of these macros do not include the delimiters.

Starting with AxoSyslog version 4.5, you can omit the columns() option, and extract the values into matches ($1, $2, $3, and so on), which are available as the anonymous list $*. For example:

@version: current

log {
    source { tcp(port(2000) flags(no-parse)); };

    parser { csv-parser(delimiters(',') dialect(escape-backslash)); };
    destination { stdout(template("$ISODATE $*\n")); };
};

delimiters()

Description: The delimiter is the character or string that separates the columns in the message. If you specify multiple characters using the delimiters(chars("<delimiter_characters>)) option, every character will be treated as a delimiter. To separate the columns at the tabulator (tab character), specify \\t. For example, to separate the text at every hyphen (-) and colon (:) character, use delimiters(chars("-:")), Note that the delimiters will not be included in the column values.

String delimiters

If you have to use a string as a delimiter, list your string delimiters in the delimiters(strings("<delimiter_string1>, "<delimiter_string2>> ...)") format.

By default, AxoSyslog uses space as a delimiter. If you want to use only the strings as delimiters, you have to disable the space delimiter, for example: delimiters(chars(""), strings("<delimiter_string>))

Otherwise, AxoSyslog will use the string delimiters in addition to the default character delimiter, so delimiters(strings("==")) actually equals delimiters(chars(" "), strings("==")), and not delimiters(chars(""), strings("=="))

Multiple delimiters

If you use more than one delimiter, note the following points:

• AxoSyslog will split the message at the nearest possible delimiter. The order of the delimiters in the configuration file does not matter.

• You can use both string delimiters and character delimiters in a parser.

• The string delimiters can include characters that are also used as character delimiters.

• If a string delimiter and a character delimiter both match at the same position of the message, AxoSyslog uses the string delimiter.

dialect()

Description: Specifies how to handle escaping in the parsed message. Default value: escape-none

   parser p_demo_parser {
        csv-parser(
            prefix(".csv.")
            delimiters(" ")
            dialect(escape-backslash)
            flags(strip-whitespace, greedy)
            columns("column1", "column2", "column3")
        );
    };

The following values are available.

• escape-backslash: The parsed message uses the backslash (\\) character to escape quote characters.
• escape-backslash-with-sequences: The parsed message uses "" as an escape character but also supports C-style escape sequences, like \n or \r. Available in AxoSyslog version 4.0 and later.
• escape-double-char: The parsed message repeats the quote character when the quote character is used literally. For example, to escape a comma (,), the message contains two commas (,,).
• escape-none: The parsed message does not use any escaping for using the quote character literally.

flags()

Description: Specifies various options for parsing the message. The following flags are available:

• drop-invalid: When the drop-invalid option is set, the parser does not process messages that do not match the parser. For example, a message does not match the parser if it has less columns than specified in the parser, or it has more columns but the greedy flag is not enabled. Using the drop-invalid option practically turns the parser into a special filter, that matches messages that have the predefined number of columns (using the specified delimiters).

  Note

  Messages dropped as invalid can be processed by a fallback log path. For details on the fallback option, see Log path flags.

• greedy: The greedy option assigns the remainder of the message to the last column, regardless of the delimiter characters set. You can use this option to process messages where the number of columns varies.

• strip-whitespace: The strip-whitespace flag removes leading and trailing whitespaces from all columns.

Example: Adding the end of the message to the last column

If the greedy option is enabled, AxoSyslog adds the not-yet-parsed part of the message to the last column, ignoring any delimiter characters that may appear in this part of the message.

For example, you receive the following comma-separated message: example 1, example2, example3, and you segment it with the following parser:

    csv-parser(columns("COLUMN1", "COLUMN2", "COLUMN3") delimiters(","));

The COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, respectively. If the message looks like example 1, example2, example3, some more information, then any text appearing after the third comma (that is, some more information) is not parsed, and possibly lost if you use only the variables to reconstruct the message (for example, to send it to different columns of an SQL table).

Using the greedy flag will assign the remainder of the message to the last column, so that the COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, some more information.

    csv-parser(columns("COLUMN1", "COLUMN2", "COLUMN3") delimiters(",") flags(greedy));

null()

Description: If the value of a column is the value of the null() parameter, AxoSyslog changes the value of the column to an empty string. For example, if the columns of the message contain the “N/A” string to represent empty values, you can use the null("N/A") option to change these values to empty stings.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

This parser does not have a default prefix. To configure a custom prefix, use the following format:

   parser {
        csv-parser(prefix("myprefix."));
    };

on-type-error()

Description: Specifies what to do when casting a parsed value to a specific data type fails. Note that the flags(drop-invalid) option and the on-error() global option also affects the behavior.

Accepts the same values as the on-error() global option.

quote-pairs()

Description: List quote-pairs between single quotes. Delimiter characters or strings enclosed between quote characters are ignored. Note that the beginning and ending quote character does not have to be identical, for example, [} can also be a quote-pair. For an example of using quote-pairs() to parse Apache log files, see Example: Parsing Apache log files.

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

For examples, see Example: Segmenting hostnames separated with a dash and Example: Segmenting a part of a message.

5 - Parsing dates and timestamps

The date parser can extract dates from non-syslog messages. It operates by default on the ${MESSAGE} part of the log message, but can operate on any template or field provided. The parsed date will be available as the sender date (that is, the ${S_DATE}, ${S_ISODATE}, ${S_MONTH}, and so on, and related macros). (To store the parsed date as the received date, use the time-stamp(recvd) option.)

Declaration

   parser parser_name {
        date-parser(
            format("<format-string-for-the-date>")
            template("<field-to-parse>")
        );
    };

Example: Using the date-parser()

In the following example, AxoSyslog parses dates like 01/Jan/2016:13:05:05 PST from a field called MY_DATE using the following format string: format("%d/%b/%Y:%H:%M:%S %Z") (how you create this field from the incoming message is not shown in the example). In the destination template every message will begin with the timestamp in ISODATE format. Since the syslog parser is disabled, AxoSyslog will include the entire original message (including the original timestamp) in the ${MESSAGE} macro.

   source s_file {
        file("/tmp/input" flags(no-parse));
    };
    
    destination d_file {
        file(
            "/tmp/output"
            template("${S_ISODATE} ${MESSAGE}\n")
        );
    };
    
    log {
        source(s_file);
        parser { date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("${MY_DATE}")); };
        destination(d_file);
    };

In the template option, you can use template functions to specify which part of the message to parse with the format string. The following example selects the first 24 characters of the ${MESSAGE} macro.

   date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("$(substr ${MESSAGE} 0 24)") );

In AxoSyslog version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:

   date-parser(format(
        "%FT%T.%f",
        "%F %T,%f",
        "%F %T"
    ));

If you need to modify or correct the timezone of the message after parsing, see Rewrite the timezone of a message.

5.1 - Options of date-parser() parsers

The date-parser() parser has the following options.

format()

Description: Specifies the format how AxoSyslog should parse the date. You can use the following format elements:

   %%      PERCENT
    %a      day of the week, abbreviated
    %A      day of the week
    %b      month abbr
    %B      month
    %c      MM/DD/YY HH:MM:SS
    %C      ctime format: Sat Nov 19 21:05:57 1994
    %d      numeric day of the month, with leading zeros (eg 01..31)
    %e      like %d, but a leading zero is replaced by a space (eg  1..31)
    %f      microseconds, leading 0's, extra digits are silently discarded
    %D      MM/DD/YY
    %G      GPS week number (weeks since January 6, 1980)
    %h      month, abbreviated
    %H      hour, 24 hour clock, leading 0's)
    %I      hour, 12 hour clock, leading 0's)
    %j      day of the year
    %k      hour
    %l      hour, 12 hour clock
    %L      month number, starting with 1
    %m      month number, starting with 01
    %M      minute, leading 0's
    %n      NEWLINE
    %o      ornate day of month -- "1st", "2nd", "25th", etc.
    %p      AM or PM
    %P      am or pm (Yes %p and %P are backwards :)
    %q      Quarter number, starting with 1
    %r      time format: 09:05:57 PM
    %R      time format: 21:05
    %s      seconds since the Epoch, UCT
    %S      seconds, leading 0's
    %t      TAB
    %T      time format: 21:05:57
    %U      week number, Sunday as first day of week
    %w      day of the week, numerically, Sunday == 0
    %W      week number, Monday as first day of week
    %x      date format: 11/19/94
    %X      time format: 21:05:57
    %y      year (2 digits)
    %Y      year (4 digits)
    %Z      timezone in ascii format (for example, PST), or in format -/+0000
    %z      timezone in ascii format (for example, PST), or in format -/+0000  (Required element)

For example, for the date 01/Jan/2016:13:05:05 PST use the following format string: format("%d/%b/%Y:%H:%M:%S %Z")

In AxoSyslog version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:

   date-parser(format(
        "%FT%T.%f",
        "%F %T,%f",
        "%F %T"
    ));

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

flags()

guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp. For example:

   date-parser(flags(guess-timezone));

time-stamp()

Description: Determines if the parsed date values are treated as sent or received date. If you use time-stamp(stamp), AxoSyslog adds the parsed date to the S_ macros (corresponding to the sent date). If you use time-stamp(recvd), AxoSyslog adds the parsed date to the R_ macros (corresponding to the received date).

time-zone()

Description: If this option is set, AxoSyslog assumes that the parsed timestamp refers to the specified timezone. The timezone set in the time-zone() option overrides any timezone information parsed from the timestamp.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

value()

Available in AxoSyslog 4.1 and later.

Description: Instruct date-parser() to store the resulting timestamp in a name-value pair specified in value(), instead of changing the timestamp value of the LogMessage.

6 - db-parser: Process message content with a pattern database (patterndb)

6.1 - Classifying log messages

The AxoSyslog application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, AxoSyslog is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example, can label the messages as user login, application crash, file transfer, and so on events.

To find the pattern that matches a particular message, AxoSyslog uses a method called longest prefix match radix tree. This means that AxoSyslog creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.

To classify a message, AxoSyslog selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.

To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimal number (for example, 1, 123, 894054, 0xFFFF, and so on). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Using pattern parsers.

The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand AxoSyslog pattens than regular expressions.

Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by AxoSyslog can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2

A regular expression matching this message from the logcheck project: Accepted (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

An AxoSyslog database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from @QSTRING:client_addr: @port @NUMBER:port:@ ssh2

For details on using pattern databases to classify log messages, see Using pattern databases.

6.1.1 - The structure of the pattern database

The pattern database is organized as follows:

• The pattern database consists of rulesets. A ruleset consists of a Program Pattern and a set of rules: the rules of a ruleset are applied to log messages if the name of the application that sent the message matches the Program Pattern of the ruleset. The name of the application (the content of the ${PROGRAM} macro) is compared to the Program Patterns of the available rulesets, and then the rules of the matching rulesets are applied to the message. (If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it.)

• The Program Pattern can be a string that specifies the name of the application or the beginning of its name (for example, to match for sendmail, the program pattern can be sendmail, or just send), and the Program Pattern can contain pattern parsers. Note that pattern parsers are completely independent from the AxoSyslog parsers used to segment messages. Additionally, every rule has a unique identifier: if a message matches a rule, the identifier of the rule is stored together with the message.

• Rules consist of a message pattern and a class. The Message Pattern is similar to the Program Pattern, but is applied to the message part of the log message (the content of the ${MESSAGE} macro). If a message pattern matches the message, the class of the rule is assigned to the message (for example, Security, Violation, and so on).

• Rules can also contain additional information about the matching messages, such as the description of the rule, an URL, name-value pairs, or free-form tags.

• Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers.

  Note

  If the ${PROGRAM} part of a message is empty, rules with an empty Program Pattern are used to classify the message.

  If the same Program Pattern is used in multiple rulesets, the rules of these rulesets are merged, and every rule is used to classify the message. Note that message patterns must be unique within the merged rulesets, but the currently only one ruleset is checked for uniqueness.

  If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it.

6.1.2 - How pattern matching works

The followings describe how patterns work. This information applies to program patterns and message patterns alike, even though message patterns are used to illustrate the procedure.

Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers. Pattern parsers attempt to parse a sequence of characters according to certain rules.

When a new message arrives, AxoSyslog attempts to classify it using the pattern database. The available patterns are organized alphabetically into a tree, and AxoSyslog inspects the message character-by-character, starting from the beginning. This approach ensures that only a small subset of the rules must be evaluated at any given step, resulting in high processing speed. Note that the speed of classifying messages is practically independent from the total number of rules.

For example, if the message begins with the Apple string, only patterns beginning with the character A are considered. In the next step, AxoSyslog selects the patterns that start with Ap, and so on, until there is no more specific pattern left. The AxoSyslog application has a strong preference for rules that match the input string completely.

Note that literal matches take precedence over pattern parser matches: if at a step there is a pattern that matches the next character with a literal, and another pattern that would match it with a parser, the pattern with the literal match is selected. Using the previous example, if at the third step there is the literal pattern Apport and a pattern parser Ap@STRING@, the Apport pattern is matched. If the literal does not match the incoming string (for example, Apple), AxoSyslog attempts to match the pattern with the parser. However, if there are two or more parsers on the same level, only the first one will be applied, even if it does not perfectly match the message.

If there are two parsers at the same level (for example, Ap@STRING@ and Ap@QSTRING@), it is random which pattern is applied (technically, the one that is loaded first). However, if the selected parser cannot parse at least one character of the message, the other parser is used. But having two different parsers at the same level is extremely rare, so the impact of this limitation is much less than it appears.

6.1.3 - Artificial ignorance

Artificial ignorance is a method used to detect anomalies. When applied to log analysis, it means that you ignore the regular, common log messages — these are the result of the regular behavior of your system, and therefore are not too concerning. However, new messages that have not appeared in the logs before can signal important events, and should be therefore investigated. “By definition, something we have never seen before is anomalous” (Marcus J. Ranum). 

The AxoSyslog application can classify messages using a pattern database: messages that do not match any pattern are classified as unknown. This provides a way to use artificial ignorance to review your log messages. You can periodically review the unknown messages — AxoSyslog can send them to a separate destination, and add patterns for them to the pattern database. By reviewing and manually classifying the unknown messages, you can iteratively classify more and more messages, until only the really anomalous messages show up as unknown.

Obviously, for this to work, a large number of message patterns are required. The radix-tree matching method used for message classification is very effective, can be performed very fast, and scales very well. Basically the time required to perform a pattern matching is independent from the number of patterns in the database. For sample pattern databases, see Downloading sample pattern databases.

6.2 - Using pattern databases

To classify messages using a pattern database, include a db-parser() statement in your syslog-ng.conf configuration file using the following syntax:

Declaration:

   parser <identifier> {
        db-parser(file("<database_filename>"));
    };

Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.

Example: Defining pattern databases

The following statement uses the database located at /opt/syslog-ng/var/db/patterndb.xml.

   parser pattern_db {
        db-parser(
            file("/opt/syslog-ng/var/db/patterndb.xml")
        );
    };

To apply the patterns on the incoming messages, include the parser in a log statement:

   log {
        source(s_all);
        parser(pattern_db);
        destination( di_messages_class);
    };

By default, AxoSyslog tries to apply the patterns to the body of the incoming messages, that is, to the value of the $MESSAGE macro. If you want to apply patterns to a specific field, or to an expression created from the log message (for example, using template functions or other parsers), use the message-template() option. For example:

   parser pattern_db {
        db-parser(
            file("/opt/syslog-ng/var/db/patterndb.xml")
            message-template("${MY-CUSTOM-FIELD-TO-PROCESS}")
        );
    };

By default, AxoSyslog uses the name of the application (content of the ${PROGRAM} macro) to select which rules to apply to the message. If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it. For example:

   parser pattern_db {
        db-parser(
            file("/opt/syslog-ng/var/db/patterndb.xml")
            program-template("${MY-CUSTOM-FIELD-TO-SELECT-RULES}")
        );
    };

Note that the program-template() option is available in AxoSyslog version 3.21 and later.

Example: Using classification results

The following destination separates the log messages into different files based on the class assigned to the pattern that matches the message (for example, Violation and Security type messages are stored in a separate file), and also adds the ID of the matching rule to the message:

   destination di_messages_class {
        file(
            "/var/log/messages-${.classifier.class}"
            template("${.classifier.rule_id};${S_UNIXTIME};${SOURCEIP};${HOST};${PROGRAM};${PID};${MESSAGE}\n")
            template-escape(no)
        );
    };

Note that if you chain pattern databases, that is, use multiple databases in the same log path, the class assigned to the message (the value of ${.classifier.class}) will be the one assigned by the last pattern database. As a result, a message might be classified as unknown even if a previous parser successfully classified it. For example, consider the following configuration:

   log {
        ...
        parser(db_parser1);
        parser(db_parser2);
        ...
    };

Even if db_parser1 matches the message, db_parser2 might set ${.classifier.class} to unknown. To avoid this problem, you can use an ‘if’ statement to apply the second parser only if the first parser could not classify the message:

   log {
        ...
        parser{ db-parser(file("db_parser1.xml")); };
        if (match("^unknown$" value(".classifier.class"))) {
            parser { db-parser(file("db_parser2.xml")); };
        };
        ...
    };

For details on how to create your own pattern databases see The pattern database format.

Drop unmatched messages

If you want to automatically drop unmatched messages (that is, discard every message that does not match a pattern in the pattern database), use the drop-unmatched() option in the definition of the pattern database:

   parser pattern_db {
        db-parser(
            file("/opt/syslog-ng/var/db/patterndb.xml")
            drop-unmatched(yes)
        );
    };

Note that the drop-unmatched() option is available in AxoSyslog version 3.11 and later.

6.2.1 - Using parser results in filters and templates

The results of message classification and parsing can be used in custom filters and templates, for example, in file and database templates. The following built-in macros allow you to use the results of the classification:

• The .classifier.class macro contains the class assigned to the message (for example, violation, security, or unknown).

• The .classifier.rule_id macro contains the identifier of the message pattern that matched the message.

• The .classifier.context_id macro contains the identifier of the context for messages that were correlated. For details on correlating messages, see Correlating log messages using pattern databases.

Example: Using classification results for filtering messages

To filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in a log statement.

   filter fi_class_violation {
        match(
            "violation"
            value(".classifier.class")
            type("string")
        );
    };

   log {
        source(s_all);
        parser(pattern_db);
        filter(fi_class_violation);
        destination(di_class_violation);
    };

Filtering on the unknown class selects messages that did not match any rule of the pattern database. Routing these messages into a separate file allows you to periodically review new or unknown messages.

To filter on messages matching a specific classification rule, create a filter that checks the .classifier.rule_id macro. The unique identifier of the rule (for example, e1e9c0d8-13bb-11de-8293-000c2922ed0a) is the id attribute of the rule in the XML database.

   filter fi_class_rule {
        match(
            "e1e9c0d8-13bb-11de-8293-000c2922ed0a"
            value(".classifier.rule_id")
            type("string")
        );
    };

Pattern database rules can assign tags to messages. These tags can be used to select tagged messages using the tags() filter function.

   filter f_tag_filter {tags(".classifier.system");};

The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.

Example: Using pattern parsers as macros

For example, you want to parse messages of an application that look like "Transaction: <type>", where <type> is a string that has different values (for example, refused, accepted, incomplete, and so on). To parse these messages, you can use the following pattern:

   'Transaction: @ESTRING::.@'

Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, for example:

   'Transaction: @ESTRING:TRANSACTIONTYPE:.@'

After that, add a custom template to the log path that uses this template. For example, to select every accepted transaction, use the following custom filter in the log path:

   match("accepted" value("TRANSACTIONTYPE"));

6.2.2 - Downloading sample pattern databases

To simplify the building of pattern databases, Axoflow has released sample databases. You can download sample pattern databases from the Axoflow GitHub page.

Note that these pattern databases are only samples and experimental databases. They are not officially supported, and may or may not work in your environment.

The pattern databases are available under the Creative Commons Attribution-Share Alike 3.0 (CC by-SA) license. This includes every pattern database written by community contributors or the Axoflow staff. It means that:

• You are free to use and modify the patterns for your needs.

• If you redistribute the pattern databases, you must distribute your modifications under the same license.

• If you redistribute the pattern databases, you must make it obvious that the source of the original pattern databases is the GitHub page.

For legal details, the full text of the license is available here.

If you create patterns that are not available in the GitHub repository, consider sharing them with the community. To do this, open a GitHub issue, or contact us.

6.2.3 - Correlating log messages using pattern databases

The AxoSyslog application can correlate log messages identified using pattern databases. Alternatively, you can also correlate log messages using the grouping-by() parser. For details, see Correlating messages using the grouping-by() parser.

Log messages are supposed to describe events, but applications often separate information about a single event into different log messages. For example, the Postfix email server logs the sender and recipient addresses into separate log messages, or in case of an unsuccessful login attempt, the OpenSSH server sends a log message about the authentication failure, and the reason of the failure in the next message. Of course, messages that are not so directly related can be correlated as well, for example, login-logout messages, and so on.

To correlate log messages with AxoSyslog, you can add messages into message-groups called contexts. A context consists of a series of log messages that are related to each other in some way, for example, the log messages of an SSH session can belong to the same context. As new messages come in, they may be added to a context. Also, when an incoming message is identified it can trigger actions to be performed, for example, generate a new message that contains all the important information that was stored previously in the context.

(For details on triggering actions and generating messages, see Triggering actions for identified messages.)

There are two attributes for pattern database rules that determine if a message matching the rule is added to a context: context-scope and context-id. The context-scope attribute acts as an early filter, selecting messages sent by the same process (${HOST}${PROGRAM}${PID} is identical), application (${HOST}${PROGRAM} is identical), or host, while the context-id actually adds the message to the context specified in the id. The context-id can be a simple string, or can contain macros or values extracted from the log messages for further filtering. Starting with AxoSyslog version 3.5, if a message is added to a context, AxoSyslog automatically adds the identifier of the context to the .classifier.context_id macro of the message.

Another parameter of a rule is the context-timeout attribute, which determines how long a context is stored, that is, how long AxoSyslog waits for related messages to arrive.

Note the following points about timeout values:

• When a new message is added to a context, AxoSyslog will restart the timeout using the context-timeout set for the new message.

• When calculating if the timeout has already expired or not, AxoSyslog uses the timestamps of the incoming messages, not system time elapsed between receiving the two messages (unless the messages do not include a timestamp, or the keep-timestamp(no) option is set). That way AxoSyslog can be used to process and correlate already existing log messages offline. However, the timestamps of the messages must be in chronological order (that is, a new message cannot be older than the one already processed), and if a message is newer than the current system time (that is, it seems to be coming from the future), AxoSyslog will replace its timestamp with the current system time.

  Example: How AxoSyslog calculates context-timeout

  Consider the following two messages:

      <38>1990-01-01T14:45:25 customhostname program6[1234]: program6 testmessage
      <38>1990-01-01T14:46:25 customhostname program6[1234]: program6 testmessage
  

  If the context-timeout is 10 seconds and AxoSyslog receives the messages within 1 second, the timeout event will occour immediately, because the difference of the two timestamp (60 seconds) is larger than the timeout value (10 seconds).

• Avoid using unnecessarily long timeout values on high-traffic systems, as storing the contexts for many messages can require considerable memory. For example, if two related messages usually arrive within seconds, it is not needed to set the timeout to several hours.

Example: Using message correlation

   <rule xml:id="..." context-id="ssh-session" context-timeout="86400" context-scope="process">
        <patterns>
            <pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern>
        </patterns>
    ...
    </rule>

For details on configuring message correlation, see the context-id, context-timeout, and context-scope attributes of pattern database rules.

6.2.3.1 - Referencing earlier messages of the context

When using the <value> element in pattern database rules together with message correlation, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For example, if there are three log messages in a context, and you are creating a generated message for the third log message, the ${HOST}@1 expression refers to the host field of the current (third) message in the context, the ${HOST}@2 expression refers to the host field of the previous (second) message in the context, ${PID}@3 to the PID of the first message, and so on. For example, the following message can be created from SSH login/logout messages (for details on generating new messages, see Triggering actions for identified messages): An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}.

Example: Referencing values from an earlier message

The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE} </value>
                </values>
            </message>
        </action>
    </actions>

If you do not know in which message of the context contains the information you need, you can use the grep, the context-lookup, or the context-values template functions.

Example: Using the grep template function

The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.

   $(grep ("${username}" == "root") ${auth_method})

To perform calculations on fields that have numerical values, see Numerical operations.

6.3 - Triggering actions for identified messages

The AxoSyslog application can generate (trigger) messages automatically if certain events occur, for example, a specific log message is received, or the correlation timeout of a message expires. Basically, you can define messages for every pattern database rule that are emitted when a message matching the rule is received. Triggering messages is often used together with message correlation, but can also be used separately. When used together with message correlation, you can also create a new correlation context when a new message is received.

The generated message is injected into the same place where the db-parser() statement is referenced in the log path. To post the generated message into the internal() source instead, use the inject-mode() option in the definition of the parser.

Example: Sending triggered messages to the internal() source

To send the generated messages to the internal source, use the inject-mode(internal) option:

   parser p_db {
        db-parser(
            file("mypatterndbfile.xml")
            inject-mode(internal)
        );
    };

To inject the generated messages where the pattern database is referenced, use the inject-mode(pass-through) option:

   parser p_db {
        db-parser(
            file("mypatterndbfile.xml")
            inject-mode(pass-through)
        );
    };

The generated message must be configured in the pattern database rule. It is possible to create an entire message, use macros and values extracted from the original message with pattern database, and so on.

Example: Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
                </values>
            </message>
        </action>
    </actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example: Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

   <actions>
        <action>
            <message inherit-properties='TRUE'>
                <values>
                    <value name="PROGRAM">overriding-original-program-name</value>
                </values>
            </message>
        </action>
    </actions>

Example: Creating a new context from an action

In AxoSyslog version 3.8 and newer, you can create a new context as an action. For details, see Element: create-context.

The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.

   <rule provider='test' id='12' class='violation'>
      <patterns>
        <pattern>simple-message-with-action-to-create-context</pattern>
      </patterns>
      <actions>
        <action trigger='match'>
          <create-context context-id='1000' context-timeout='60' context-scope='program'>
            <message inherit-properties='context'>
              <values>
                <value name='MESSAGE'>context message</value>
              </values>
            </message>
          </create-context>
        </action>
      </actions>
    </rule>

For details on configuring actions, see the description of the pattern database format.

6.3.1 - Conditional actions

To limit when a message is triggered, use the condition attribute and specify a filter expression: the action will be executed only if the condition is met. For example, the following action is executed only if the message was sent by the host called myhost.

   <action condition="'${HOST}' == 'myhost'">

You can use the same operators in the condition that can be used in filters. For details, see Comparing macro values in filters.

The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 ${DATE} </value>
                </values>
            </message>
        </action>
    </actions>

Example: Actions based on the number of messages

The following example triggers different actions based on the number of messages in the context. This way you can check if the context contains enough messages for the event to be complete, and execute a different action if it does not.

   <actions>
        <action condition='"$(context-length)" >= "4"'>
            <message>
                <values>
                    <value name="PROGRAM">event</value>
                    <value name="MESSAGE">Event complete</value>
                </values>
            </message>
        </action>
        <action condition='"$(context-length)" < "4"'>
            <message>
                <values>
                    <value name="PROGRAM">error</value>
                <value name="MESSAGE">Error detected</value>
                </values>
            </message>
        </action>
    </actions>

6.3.2 - External actions

To perform an external action when a message is triggered, for example, to send the message in an email, you have to route the generated messages to an external application using the program() destination.

Example: Sending triggered messages to external applications

The following sample configuration selects the triggered messages and sends them to an external script.

1. Set a field in the triggered message that is easy to identify and filter. For example:

       <values>
           <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
           <value name="TRIGGER">yes</value>
       </values>
   

2. Create a destination that will process the triggered messages.

       destination d_triggers {
           program("/bin/myscript"; );
       };
   

3. Create a filter that selects the triggered messages from the internal source.

       filter f_triggers {
           match("yes" value ("TRIGGER") type(string));
       };
   

4. Create a logpath that selects the triggered messages from the internal source and sends them to the script:

       log { source(s_local); filter(f_triggers); destination(d_triggers); };
   

5. Create a script that will actually process the generated messages, for example:

       #!/usr/bin/perl
       while (<>) {
           # body of the script to send emails, snmp traps, and so on
       }
   

6.3.3 - Actions and message correlation

Certain features of generating messages can be used only if message correlation is used as well. For details on correlating messages, see Correlating log messages using pattern databases.

• The AxoSyslog application automatically fills the fields for the generated message based on the scope of the context, for example, the HOST and PROGRAM fields if the context-scope is program.

• When used together with message correlation, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For details, see Referencing earlier messages of the context.

  Example: Referencing values from an earlier message

  The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):

     <actions>
          <action>
              <message>
                  <values>
                      <value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE} </value>
                  </values>
              </message>
          </action>
      </actions>
  

• You can use the name-value pairs of other messages of the context. If you set the inherit-properties attribute of the generated message to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. This means that you can refer to a name-value pair without having to know which message of the context included it. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. To refer to an earlier value, use the @<distance-of-referenced-message-from-the-current> suffix format.

      <action>
          <message inherit-properties='context'>
  

  Example: Using the inherit-properties option

  For example, if inherit-properties is set to context, and you have a rule that collects SSH login and logout messages to the same context, you can use the following value to generate a message collecting the most important information form both messages, including the beginning and end date.

      <value name="MESSAGE">An SSH session for ${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid: $PID.</value>
  

  The following is a detailed rule for this purpose.

      <patterndb version='4' pub_date='2015-04-13'>
          <ruleset name='sshd' id='12345678'>
              <pattern>sshd</pattern>
                  <rules>
                      <!-- The pattern database rule for the first log message -->
                      <rule provider='me' id='12347598' class='system'
                          context-id="ssh-login-logout" context-timeout="86400"
                          context-scope="process">
                      <!-- Note the context-id that groups together the
                      relevant messages, and the context-timeout value that
                      determines how long a new message can be added to the
                      context  -->
                          <patterns>
                              <pattern>Accepted @ESTRING:SSH.AUTH_METHOD: @for @ESTRING:SSH_USERNAME: @from @ESTRING:SSH_CLIENT_ADDRESS: @port @ESTRING:: @@ANYSTRING:SSH_SERVICE@</pattern>
                              <!-- This is the actual pattern used to identify
                              the log message. The segments between the @
                              characters are parsers that recognize the variable
                              parts of the message - they can also be used as
                              macros.  -->
                          </patterns>
                      </rule>
                      <!-- The pattern database rule for the fourth log message -->
                      <rule provider='me' id='12347599' class='system' context-id="ssh-login-logout" context-scope="process">
                          <patterns>
                               <pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:SSH_USERNAME@</pattern>
                          </patterns>
                          <actions>
                              <action>
                                  <message inherit-properties='context'>
                                      <values>
                                          <value name="MESSAGE">An SSH session for ${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid: $PID.</value>
                                          <value name="TRIGGER">yes</value>
                                          <!-- This is the new log message
                                          that is generated when the logout
                                          message is received. The macros ending
                                          with @2 reference values of the
                                          previous message from the context. -->
                                      </values>
                                  </message>
                              </action>
                          </actions>
                      </rule>
                  </rules>
          </ruleset>
      </patterndb>
  

• It is possible to generate a message when the context-timeout of the original message expires and no new message is added to the context during this time. To accomplish this, include the trigger="timeout" attribute in the action element:

      <action trigger="timeout">
  

  Example: Sending alert when a client disappears

  The following example shows how to combine various features of AxoSyslog to send an email alert if a client stops sending messages.

  • Configure your clients to send MARK messages periodically. It is enough to configure MARK messages for the destination that forwards your log messages to your AxoSyslog server (mark-mode(periodical)).

  • On your AxoSyslog server, create a pattern database rule that matches on the incoming MARK messages. In the rule, set the context-scope attribute to host, and the context-timeout attribute to a value that is higher than the mark-freq value set on your clients (by default, mark-freq is 1200 seconds, so set context-timeout at least to 1500 seconds, but you might want to use a higher value, depending on your environment).

  • Add an action to this rule that sends you an email alert if the context-timeout expires, and the server does not receive a new MARK message (<action trigger="timeout">).

  • On your AxoSyslog server, use the pattern database in the log path that handles incoming log messages.

6.4 - Creating pattern databases

6.4.1 - Using pattern parsers

Pattern parsers attempt to parse a part of the message using rules specific to the type of the parser. Parsers are enclosed between @ characters. The syntax of parsers is the following:

• a beginning @ character,

• the type of the parser written in capitals,

• optionally a name,

• parameters of the parser, if any, and

• a closing @ character.

Example: Pattern parser syntax

A simple parser:

   @STRING@

A named parser:

   @STRING:myparser_name@

A named parser with a parameter:

   @STRING:myparser_name:*@

A parser with a parameter, but without a name:

   @STRING::*@

Patterns and literals can be mixed together. For example, to parse a message that begins with the Host: string followed by an IP address (for example, Host: 192.168.1.1), the following pattern can be used: Host:@IPv4@.

Example: Using the STRING and ESTRING parsers

For example, look at the following message: user=joe96 group=somegroup.

• @STRING:mytext:@ parses only to the first non-alphanumeric character (=), parsing only user, so the value of the ${mytext} macro will be user

• @STRING:mytext:=@ parses the equation mark as well, and proceeds to the next non-alphanumeric character (the whitespace), resulting in user=joe96

• @STRING:mytext:= @ will parse the whitespace as well, and proceed to the next non-alphanumeric non-equation mark non-whitespace character, resulting in user=joe96 group=somegroup

Of course, usually it is better to parse the different values separately, like this: "user=@STRING:user@ group=@STRING:group@".

If the username or the group may contain non-alphanumeric characters, you can either include these in the second parameter of the parser (as shown at the beginning of this example), or use an ESTRING parser to parse the message till the next whitespace: "user=@ESTRING:user: @group=@ESTRING:group: @".

6.4.1.1 - Pattern parsers

The following parsers are available in AxoSyslog. The internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair. For details on data types, see Specifying data types in value-pairs.

@ANYSTRING@

Parses everything to the end of the message, you can use it to collect everything that is not parsed specifically to a single macro. In that sense its behavior is similar to the greedy() option of the CSV parser.

@DOUBLE@

An obsolete alias of the @FLOAT@ parser.

@EMAIL@

This parser matches an email address. The parameter is a set of characters to strip from the beginning and the end of the email address. That way email addresses enclosed between other characters can be matched easily (for example, <[email protected]> or "[email protected]". Characters that are valid for a hostname are not stripped from the end of the hostname. This includes a trailing period if present.

For example, the @EMAIL:email:"[<]> parser will match any of the following email addresses: <[email protected]>, [[email protected]], "[email protected]", and set the value of the email macro to [email protected].

@ESTRING@

This parser has a required parameter that acts as the stopcharacter: the parser parses everything until it finds the stopcharacter. For example, to stop by the next " (double quote) character, use @ESTRING::"@. You can use the colon (:) as stopcharacter as well, for example: @ESTRING:::@. You can also specify a stopstring instead of a single character, for example, @ESTRING::stop_here.@. The @ character cannot be a stopcharacter, nor can line-breaks or tabs.

@FLOAT@

A floating-point number that may contain a dot (.) character. (Up to AxoSyslog 3.1, the name of this parser was @DOUBLE@.)

@HOSTNAME@

Parses a generic hostname. The hostname may contain only alphanumeric characters (A-Z,a-z,0-9), hypen (-), or dot (.).

@IPv4@

Parses an IPv4 IP address (numbers separated with a maximum of 3 dots).

@IPv6@

Parses any valid IPv6 IP address.

@IPvANY@

Parses any IP address.

@LLADDR@

Parses a Link Layer Address in the xx:xx:xx:... form, where each xx is a 2 digit HEX number (an octet). The parameter specifies the maximum number of octets to match and defaults to 20. The MACADDR parser is a special wrapper using the LLADDR parser. For example, the following parser parses maximally 10 octets, and stores the results in the link-level-address macro:

   @LLADDR:link-level-address:10@

@MACADDR@

Parses the standard format of a MAC-48 address, consisting of is six groups of two hexadecimal digits, separated by colons. For example, 00:50:fc:e3:cd:37.

@NLSTRING@

This parser parses everything until the next new-line character (more precisely, until the next Unix-style LF or Windows-style CRLF character). For single-line messages, NLSTRING is equivalent with ANYSTRING. For multi-line messages, NLSTRING parses to the end of the current line, while ANYSTRING parses to the end of the message. Using NLSTRING is useful when parsing multi-line messages, for example, Windows logs. For example, the following pattern parses information from Windows security auditing logs.

   <pattern>Example-PC\Example: Security Microsoft Windows security auditing.: [Success Audit] A new process has been created.
    
        Subject:
        Security ID: @LNSTRING:.winaudit.SubjectUserSid@
        Account Name: @LNSTRING:.winaudit.SubjectUserName@
        Account Domain: @LNSTRING:.winaudit.SubjectDomainName@
        Logon ID: @LNSTRING:.winaudit.SubjectLogonId@
    
        Process Information:
        New Process ID: @LNSTRING:.winaudit.NewProcessId@
        New Process Name: @LNSTRING:.winaudit.NewProcessName@
        Token Elevation Type: @LNSTRING:.winaudit.TokenElevationType@
        Creator Process ID: @LNSTRING:.winaudit.ProcessId@
        Process Command Line: @LNSTRING:.winaudit.CommandLine@
    
        Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.</pattern>

@NUMBER@

A sequence of decimal (0-9) numbers (for example, 1, 0687, and so on). Note that if the number starts with the 0x characters, it is parsed as a hexadecimal number, but only if at least one valid character follows 0x. A leading hyphen (-) is accepted for non-hexadecimal numbers, but other separator characters (for example, dot or comma) are not. To parse floating-point numbers, use the @FLOAT@ parser.

@OPTIONALSET@

Parse any combination of the specified characters. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings (for example, log messages of the Squid application have whitespace padding after the username).

For example, the @OPTIONALSET:: "@ parser will parse any combination of whitespaces and double-quotes.

Available in 3.31 and later.

@PCRE@

Use Perl-Compatible Regular Expressions (as implemented by the PCRE library), after the identification of the potential patterns has happened by the radix implementation.

Syntax: @PCRE:name:regexp@

@QSTRING@

Parse a string between the quote characters specified as parameter. Note that the quote character can be different at the beginning and the end of the quote, for example: @QSTRING::"@ parses everything between two quotation marks ("), while @QSTRING:\<\>@ parses from an opening bracket to the closing bracket. The @ character cannot be a quote character, nor can line-breaks or tabs.

@SET@

Parse any combination of the specified characters until another character is found. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings (for example, log messages of the Squid application have whitespace padding after the username).

For example, the @SET:: "@ parser will parse any combination of whitespaces and double-quotes.

Available in AxoSyslog 3.4 and later.

@STRING@

A sequence of alphanumeric characters (0-9, A-z), not including any whitespace. Optionally, other accepted characters can be listed as parameters (for example, to parse a complete sentence, add the whitespace as parameter, like: @STRING:: @). Note that the @ character cannot be a parameter, nor can line-breaks or tabs.

6.4.2 - What's new in the pattern database format V5

The V5 database format has the following differences compared to the V4 format:

• The <ruleset> element can now store multiple reference URLs using the new <rule_urls> element. For details, see Element: ruleset.

• In an <action>, you can now initialize a new context. As a result, the <message> element is not required. For details, see Element: create-context.

• The inherit-properties attribute is deprecated, use the inherit-mode attribute instead. For details, see Element: action.

6.4.3 - The pattern database format

Pattern databases are XML files that contain rules describing the message patterns. For sample pattern databases, see Downloading sample pattern databases.

The following scheme describes the V5 format of the pattern database. This format is backwards-compatible with the earlier formats.

For a sample database containing only a single pattern, see Example: A pattern database containing a single rule.

Example: A pattern database containing a single rule

The following pattern database contains a single rule that matches a log message of the ssh application. A sample log message looks like:

   Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2

The following is a simple pattern database containing a matching rule.

   <patterndb version='5' pub_date='2010-10-17'>
        <ruleset name='ssh' id='123456678'>
            <pattern>ssh</pattern>
                <rules>
                    <rule provider='me' id='182437592347598' class='system'>
                        <patterns>
                            <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
                        </patterns>
                    </rule>
                </rules>
        </ruleset>
    </patterndb>

Note that the rule uses macros that refer to parts of the message, for example, you can use the ${SSH_USERNAME} macro refer to the username used in the connection.

The following is the same example, but with a test message and test values for the parsers.

   <patterndb version='4' pub_date='2010-10-17'>
        <ruleset name='ssh' id='123456678'>
            <pattern>ssh</pattern>
                <rules>
                    <rule provider='me' id='182437592347598' class='system'>
                        <patterns>
                            <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
                        </patterns>
                        <examples>
                            <example>
                                <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
                                <test_values>
                                    <test_value name="SSH.AUTH_METHOD">password</test_value>
                                    <test_value name="SSH_USERNAME">sampleuser</test_value>
                                    <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                                    <test_value name="SSH_PORT_NUMBER">42156</test_value>
                                </test_values>
                           </example>
                        </examples>
                    </rule>
                </rules>
        </ruleset>
    </patterndb>

6.4.3.1 - Element: patterndb

Location

/patterndb

Description

The container element of the pattern database.

Attributes

• version: The schema version of the pattern database. The current version is 4.

• pubdate: The publication date of the XML file.

Children

• ruleset

Example

   <patterndb version='4' pub_date='2010-10-25'>

6.4.3.2 - Element: ruleset

Location

/patterndb/ruleset

Description

A container element to group log patterns for an application or program. A <patterndb> element may contain any number of <ruleset> elements.

Attributes

• name: The name of the application. Note that the function of this attribute is to make the database more readable, syslog-ng uses the <pattern> element to identify the applications sending log messages.

• id: A unique ID of the application, for example, the md5 sum of the name attribute.

Children

• patterns

• rules

• actions

• tags

• description: OPTIONAL — A description of the ruleset or the application.

• url: OPTIONAL — An URL referring to further information about the ruleset or the application.

• rule_urls: OPTIONAL — To list multiple URLs referring to further information about the ruleset or the application, enclose the <url> elements into an <urls> element.

Example

   <ruleset name='su' id='480de478-d4a6-4a7f-bea4-0c0245d361e1'>

6.4.3.3 - Element: patterns

Location

/patterndb/ruleset/patterns

Description

A container element. A <patterns> element may contain any number of `elements.

Attributes

N/A

Children

• pattern: The name of the application — syslog-ng matches this value to the ${PROGRAM} header of the syslog message to find the rulesets applicable to the syslog message.

  Specifying multiple patterns is useful if two or more applications have different names (that is, different ${PROGRAM} fields), but otherwise send identical log messages.

  It is not necessary to use multiple patterns if only the end of the ${PROGRAM} fields is different, use only the beginning of the ${PROGRAM} field as the pattern. For example, the Postfix email server sends messages using different process names, but all of them begin with the postfix string.

  You can also use parsers in the program pattern if needed, and use the parsed results later. For example: `stfix\@ESTRING:.postfix.component:[@»

  Note

  If the `element of a ruleset is not specified, AxoSyslog will use this ruleset as a fallback ruleset: it will apply the ruleset to messages that have an empty PROGRAM header, or if none of the program patterns matched the PROGRAM header of the incoming message.

Example

   <patterns>
        <pattern>firstapplication</pattern>
        <pattern>otherapplication</pattern>
    </patterns>

Using parsers in the program pattern:

   <pattern>postfix\@ESTRING:.postfix.component:[@</pattern>

6.4.3.4 - Element: rules

Location

/patterndb/ruleset/rules

Description

A container element for the rules of the ruleset.

Attributes

N/A

Children

• rule

Example

   <rules>
        <rule provider='me' id='182437592347598' class='system'>
            <patterns>
                <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
            </patterns>
        </rule>
    </rules>

6.4.3.5 - Element: rule

Location

/patterndb/ruleset/rules/rule

Description

An element containing message patterns and how a message that matches these patterns is classified.

The element may contain any number of elements.

Attributes

• provider: The provider of the rule. This is used to distinguish between who supplied the rule, that is, if it has been created by Axoflow, or added to the XML by a local user.

• id: The globally unique ID of the rule.

• class: The class of the rule — this class is assigned to the messages matching a pattern of this rule.

Children

• patterns

Example

   <rule provider='example' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation'>

The following example specifies attributes for correlating messages as well. For details on correlating messages, see Correlating log messages using pattern databases.

   <rule provider='example' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation' context-id='same-session' context-scope='process' context-timeout='360'>

6.4.3.6 - Element: patterns

Location

/patterndb/ruleset/rules/rule/patterns

Description

An element containing the patterns of the rule. If a element contains multiple *lements, the class of the *»>ssigned to every syslog message matching any of the patterns.

Attributes

N/A

Children

• pattern: A pattern describing a log message. This element is also called message pattern. For example:

      <pattern>+ ??? root-</pattern>
  

  Note

  Support for XML entities is limited, you can use only the following entities: \& \< \> \" \'. User-defined entities are not supported.

• description: OPTIONAL — A description of the pattern or the log message matching the pattern.

• urls

• values

• examples

Example

   <patterns>
        <pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
    </patterns>

6.4.3.7 - Element: urls

Location

/patterndb/ruleset/rules/rule/patterns/urls

Description

OPTIONAL — An element containing one or more URLs referring to further information about the patterns or the matching log messages.

Attributes

N/A

Children

• url: OPTIONAL — An URL referring to further information about the patterns or the matching log messages.

Example

N/A

6.4.3.8 - Element: values

Location

/patterndb/ruleset/rules/rule/patterns/values

Description

OPTIONAL — Name-value pairs that are assigned to messages matching the patterns, for example, the representation of the event in the message according to the Common Event Format (CEF) or Common Event Exchange (CEE). The names can be used as macros to reference the assigned values.

Attributes

N/A

Children

• value: OPTIONAL — Contains the value of the name-value pair that is assigned to the message.

  The <value> element of name-value pairs can include template functions. For details, see Using template functions, for examples, see if.

  You can associate types with values using the "type" attribute, for example, integer is a type-cast that associates $foobar with an integer type. For details on data types, see Specifying data types in value-pairs.

  <value name="foobar" type="integer">$PID</value>
  

  db-parser()’s internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair.

  When used together with message correlation, the <value> element of name-value pairs can include references to the values of earlier messages from the same context. For details, see Correlating log messages using pattern databases.

• name: The name of the name-value pair. It can also be used as a macro to reference the assigned value.

Example

   <values>
        <value name=".classifier.outcome">/Success</value>
    </values>

6.4.3.9 - Element: actions

Location

/patterndb/ruleset/actions

Description

OPTIONAL — A container element for actions that are performed if a message is recognized by the pattern. For details on actions, see Triggering actions for identified messages.

Attributes

N/A

Children

• action

Example

Example: Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
                </values>
            </message>
        </action>
    </actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example: Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

   <actions>
        <action>
            <message inherit-properties='TRUE'>
                <values>
                    <value name="PROGRAM">overriding-original-program-name</value>
                </values>
            </message>
        </action>
    </actions>

6.4.3.10 - Element: action

Location

/patterndb/ruleset/actions/action

Description

OPTIONAL — A container element describing an action that is performed when a message matching the rule is received.

Attributes

• condition: An AxoSyslog filter expression. The action is performed only if the message matches the filter. The filter can include macros and name-value pairs extracted from the message. When using actions together with message-correlation, you can also use the $(context-length) macro, which returns the number of messages in the current context. For example, this can be used to determine if the expected number of messages has arrived to the context: condition='"$(context-length)" > "5"'

• rate: Specifies maximum how many messages should be generated in the specified time period in the following format: <number-of-messages>/<period-in-seconds>. For example: 1/60 allows 1 message per minute. Rates apply within the scope of the context, that is, if context-scope="host" and rate="1/60", then maximum one message is generated per minute for every host that sends a log message matching the rule. Excess messages are dropped. Note that when applying the rate to the generated messages, AxoSyslog uses the timestamps of the log messages, similarly to calculating the context-timeout. That way rate is applied correctly even if the log messages are processed offline.

• trigger: Specifies when the action is executed. The trigger attribute has the following possible values:

  • match: Execute the action immediately when a message matching the rule is received.

  • timeout: Execute the action when the correlation timer (context-timeout) of the pattern database rule expires. This is available only if actions are used together with correlating messages.

Children

• create-context

• message: A container element storing the message to be sent when the action is executed. Currently AxoSyslog sends these messages to the internal() destination.

  • For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

    inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.

    • context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

    • last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.

    • none: An empty message is created, without inheriting any tags or name-value pairs.

    This option is available in AxoSyslog 3.8 and later.

  • inherit-properties: This attribute is deprecated. Use the inherit-mode attribute instead.

    If set to TRUE, the original message that triggered the action is cloned, including its name-value pairs and tags.

    If set to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

    For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

    This option is available in AxoSyslog 5.3.2 and later.

• values: A container element for values and fields that are used to create the message generated by the action.

  • value: Sets the value of the message field specified in the name attribute of the element. For example, to specify the body of the generated message, use the following syntax:

    
        <value name="MESSAGE">A log message matched rule number $.classifier.rule_id</value>
    

    Note that currently it is not possible to add DATE, FACILITY, or SEVERITY fields to the message.

    When the action is used together with message correlation, the AxoSyslog application automatically adds fields to the message based on the context-scope parameter. For example, using context-scope="process" automatically fills the HOST, PROGRAM, and PID fields of the generated message.

  • name: Name of the message field set by the value element.

Example: Generating messages for pattern database matches

When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.

   <actions>
        <action>
            <message>
                <values>
                    <value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
                </values>
            </message>
        </action>
    </actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.

Example: Generating messages with inherited values

The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name

   <actions>
        <action>
            <message inherit-properties='TRUE'>
                <values>
                    <value name="PROGRAM">overriding-original-program-name</value>
                </values>
            </message>
        </action>
    </actions>

6.4.3.11 - Element: create-context

Location

/patterndb/ruleset/actions/action/create-context

Description

OPTIONAL — Creates a new correlation context from the current message and its associated context. This can be used to “split” a context.

Available in AxoSyslog version 3.8 and later.

Attributes

Children

• message: A container element storing the message that is added to the new context when the action is executed.

  • inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.

    • context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.

    • last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.

    • none: An empty message is created, without inheriting any tags or name-value pairs.

    For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages

Example

The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.

   <rule provider='test' id='12' class='violation'>
      <patterns>
        <pattern>simple-message-with-action-to-create-context</pattern>
      </patterns>
      <actions>
        <action trigger='match'>
          <create-context context-id='1000' context-timeout='60' context-scope='program'>
            <message inherit-properties='context'>
              <values>
                <value name='MESSAGE'>context message</value>
              </values>
            </message>
          </create-context>
        </action>
      </actions>
    </rule>

6.4.3.12 - Element: tags

Location

/patterndb/ruleset/tags

Description

OPTIONAL — An element containing custom keywords (tags) about the messages matching the patterns. The tags can be used to label specific events (for example, user logons). It is also possible to filter on these tags later (for details, see Tagging messages). Starting with AxoSyslog 3.2, the list of tags assigned to a message can be referenced with the ${TAGS} macro.

Attributes

N/A

Children

• tag: OPTIONAL — A keyword or tags applied to messages matching the rule.

Example

   <tags><tag>UserLogin</tag></tags>

6.4.3.13 - Element: example

Location

/patterndb/ruleset/rules/rule/patterns/examples/example

Description

OPTIONAL — A container element for a sample log message.

Attributes

N/A

Children

• test_message: OPTIONAL — A sample log message that should match this pattern. For example:

      <test_message program="myapplication">Content filter has been enabled</test_message>
  

  • program: The program pattern of the test message. For example:

        <test_message program="proftpd">ubuntu (::ffff:192.168.2.179[::ffff:192.168.2.179]) - FTP session closed.</test_message>
    

• test_values: OPTIONAL — A container element to test the results of the parsers used in the pattern.

  • test_value: OPTIONAL — The expected value of the parser when matching the pattern to the test message. For example:

        <test_value name=".dict.ContentFilter" type="string">enabled</test_value>
    

    • name: The name of the name-value pair to test.
    • type: The type of the name-value pair, one of the recognized syslog-ng type hints Specifying data types in value-pairs

Example

   <examples>
        <example>
            <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
            <test_values>
                <test_value name="SSH_AUTH_METHOD">password</test_value>
                <test_value name="SSH_USERNAME">sampleuser</test_value>
                <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                <test_value name="SSH_PORT_NUMBER" type="integer">42156</test_value>
            </test_values>
        </example>
    </examples>

6.4.3.14 - Element: examples

Location

/patterndb/ruleset/rules/rule/patterns/examples

Description

OPTIONAL — A container element for sample log messages that should be recognized by the pattern. These messages can be used also to test the patterns and the parsers.

Attributes

N/A

Children

• example

Example

   <examples>
        <example>
            <test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
            <test_values>
                <test_value name="SSH.AUTH_METHOD">password</test_value>
                <test_value name="SSH_USERNAME">sampleuser</test_value>
                <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                <test_value name="SSH_PORT_NUMBER">42156</test_value>
            </test_values>
        </example>
    </examples>

7 - Parsing enterprise-wide message model (EWMM) messages

The ewmm-parser() can be used to parse messages sent by another AxoSyslog host using the enterprise-wide message model (EWMM) format. Available in version 3.16 and later. Note that usually you do not have to use this parser directly, because the default-network-drivers() source automatically parses such messages.

Declaration:

   parser parser_name {
        ewmm-parser();
    };

8 - Fortigate parser

The Fortigate parser can parse the log messages of FortiGate/FortiOS (Fortigate Next-Generation Firewall (NGFW)). These messages do not completely comply with the syslog RFCs, making them difficult to parse. The fortigate-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs, see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:

   <PRI><NAME=VALUE PAIRS>

For example:

   <189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683510055 tz="+0300" srcip=91.234.154.139 srcname="91.234.154.139" srcport=45295 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413215 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="udp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"

If you find a message that the fortigate-parser() cannot properly parse, contact us, so we can improve the parser.

By default, the Fortigate-specific fields are extracted into name-value pairs prefixed with .fortigate. For example, the devname in the previous message becomes ${.fortigate.devname}. You can change the prefix using the prefix option of the parser.

Declaration:

   @version: 4.5.0
    @include "scl.conf"
    log {
        source { network(transport("udp") flags(no-parse)); };
        parser { fortigate-parser(); };
        destination { ... };
    };

Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.

The fortigate-parser() is actually a reusable configuration snippet configured to parse Fortigate messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, websense-parser() uses the .websense. prefix. To modify it, use the following format:

   parser {
        websense-parser(prefix("myprefix."));
    };

8.1 - Fortigate parser options

The fortigate-parser() has the following options:

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, fortigate-parser() uses the .fortigate. prefix. To modify it, use the following format:

   parser {
        fortigate-parser(prefix("myprefix."));
    };

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

9 - group-lines parser

Available in AxoSyslog version 4.2 and newer.

The group-lines() parser correlates multi-line messages received as separate, but subsequent lines into a single log message. AxoSyslog first collects the received messages into streams of related messages (based on the key() parameter), then grouped into correlation contexts up to timeout() seconds long. Multi-line messages are then identified within these contexts.

  group-lines(
    key("$FILE_NAME")
    multi-line-mode("smart")
    template("$MESSAGE")
    timeout(10)
    line-separator("\n")
  );

The parser has the following options.

key()

Description: Specifies a template that determines which messages form a single stream. Messages where the template expansion results in the same key are considered part of the same stream. Using the key() option, you can extract multi-line messages even if different streams are interleaved in your input.

line-separator()

Description: In case a multi-line message is found, this string is inserted between the of the new multi-line message. Defaults to the newline character.

multi-line-garbage()

Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.

To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.

multi-line-mode()

Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:

• indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.

      source s_tomcat {
          file("/var/log/tomcat/xxx.log" multi-line-mode(indented));
      };
  

• prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.

• prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.

  The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.

• smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.

  smart mode is available in AxoSyslog version 4.2 and newer.

  The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.

multi-line-prefix()

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.

Example: Processing Tomcat logs

The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

   2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    SEVERE: Catalina.start:
    LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use null:8080
           at org.apache.catalina.connector.Connector.start(Connector.java:1138)
           at org.apache.catalina.core.StandardService.start(StandardService.java:531)
           at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
           at org.apache.catalina.startup.Catalina.start(Catalina.java:583)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)
    2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
    INFO: Server startup in 1206 ms
    2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
    INFO: Pausing Coyote HTTP/1.1 on http-8080
    2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
    INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:

source s_file{file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0) multi-line-mode(regexp) multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));};
    };

Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.

multi-line-suffix()

Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.

To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.

scope()

Description: Specifies which messages belong to the same context. The following values are available:

• process: Only messages that are generated by the same process of a client belong to the same context, that is, messages that have identical ${HOST}, ${PROGRAM} and ${PID} values.
• program: Messages that are generated by the same application of a client belong to the same context, that is, messages that have identical ${HOST} and ${PROGRAM} values.
• host: Every message generated by a client belongs to the same context, only the ${HOST} value of the messages must be identical.
• global: Every message belongs to the same context. This is the default value.

template()

Description: A template string that specifies what constitutes an line to group-lines(). In simple cases this is ${MSG} or ${RAWMSG}.

10 - iptables parser

The iptables parser can parse the log messages of the iptables command. Available in version 3.16 and later.

Declaration:

   @version: 4.5
    @include "scl.conf"
    log {
        source { system(); };
        parser { iptables-parser(); };
        destination { ... };
    };

The iptables-parser() is actually a reusable configuration snippet configured to parse iptables messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, iptables-parser() uses the .iptables. prefix. To modify it, use the following format:

   parser { 
        iptables-parser(prefix("myprefix.")); 
    };

11 - JSON parser

JavaScript Object Notation (JSON) is a text-based open standard designed for human-readable data interchange. It is used primarily to transmit data between a server and web application, serving as an alternative to XML. It is described in RFC 4627. The AxoSyslog application can separate parts of incoming JSON-encoded log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs.

You can refer to the separated parts of the JSON message using the key of the JSON object as a macro. For example, if the JSON contains {"KEY1":"value1","KEY2":"value2"}, you can refer to the values as ${KEY1} and ${KEY2}. If the JSON content is structured, AxoSyslog converts it to dot-notation-format. For example, to access the value of the following structure {"KEY1": {"KEY2": "VALUE"}}, use the ${KEY1.KEY2} macro.

To create a JSON parser, define a parser that has the json-parser() option. Defining the prefix and the marker are optional. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message with the JSON parser, use the template() option. You can also define the parser inline in the log path.

Declaration:

   parser parser_name {
        json-parser(
            marker()
            prefix()
        );
    };

Example: Using a JSON parser

In the following example, the source is a JSON encoded log message. The syslog parser is disabled, so that AxoSyslog does not parse the message: flags(no-parse). The json-parser inserts “.json.” prefix before all extracted name-value pairs. The destination is a file that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.

   source s_json {
        network(
            port(21514
            flags(no-parse)
        );
    };
    
    destination d_json {
        file(
            "/tmp/test.json"
            template("$(format-json --scope dot-nv-pairs)\n")
        );
    };
    
    parser p_json {
        json-parser (prefix(".json."));
    };
    
    log {
        source(s_json);
        parser(p_json);
        destination(d_json);
    };

You can also define the parser inline in the log path.

   source s_json {
        network(
            port(21514)
            flags(no-parse)
        );
    };
    
    destination d_json {
        file(
            "/tmp/test.json"
            template("$(format-json --scope dot-nv-pairs)\n")
        );
    };
    
    log {
        source(s_json);
        parser {
            json-parser (prefix(".json."));
        };
        destination(d_json);
    };

11.1 - Options of JSON parsers

The JSON parser has the following options.

extract-prefix()

Description: Extract only the specified subtree from the JSON message. Use the dot-notation to specify the subtree. The rest of the message will be ignored. For example, assuming that the incoming object is named msg, the json-parser(extract-prefix("foo.bar[5]")); parser is equivalent to the msg.foo.bar[5] javascript code. Note that the resulting expression must be a JSON object in order to extract its members into name-value pairs.

This feature also works when the top-level object is an array, because you can use an array index at the first indirection level, for example: json-parser(extract-prefix("[5]")), which is equivalent to msg[5].

In addition to alphanumeric characters, the key of the JSON object can contain the following characters: \!"#$%&'()\*+,-/:;<=>?@\\^_{|}~`

It cannot contain the following characters: .[]

Example: Convert logstash eventlog format v0 to v1

The following parser converts messages in the logstash eventlog v0 format to the v1 format.

   parser p_jsoneventv0 {
        channel {
            parser {
                json-parser(extract-prefix("@fields"));
            };
            parser {
                json-parser(prefix(".json."));
            };
            rewrite {
                set("1" value("@version"));
                set("${.json.@timestamp}" value("@timestamp"));
                set("${.json.@message}" value("message"));
            };
        };
    };

key-delimiter()

Description: The delimiter character to use when parsing flattened keys. Supports Only single characters.

marker

Description: Use a marker in case of mixed log messages, to identify JSON encoded messages for the parser.

Some logging implementations require a marker to be set before the JSON payload. The JSON parser is able to find these markers and parse the message only if it is present.

Example: Using the marker option in JSON parser

This json parser parses log messages which use the “@cee:” marker in front of the json payload. It inserts “.cee.” in front of the name of name-value pairs, so later on it is easier to find name-value pairs that were parsed using this parser. (For details on selecting name-value pairs, see value-pairs().)

   parser {
            json-parser(
                marker("@cee:")
                prefix(".cee.")
            );
        };

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

This parser does not have a default prefix. To configure a custom prefix, use the following format:

   parser {
        json-parser(prefix("myprefix."));
    };

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

12 - Parsing key=value pairs

The AxoSyslog application can separate a message consisting of whitespace or comma-separated key=value pairs (for example, Postfix log messages) into name-value pairs. You can also specify other separator character instead of the equal sign, for example, colon (:) to parse MySQL log messages. The AxoSyslog application automatically trims any leading or trailing whitespace characters from the keys and values, and also parses values that contain unquoted whitespace. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs.

You can refer to the separated parts of the message using the key of the value as a macro. For example, if the message contains KEY1=value1,KEY2=value2, you can refer to the values as ${KEY1} and ${KEY2}.

To parse key=value pairs, define a parser that has the kv-parser() option. Defining the prefix is optional. By default, the parser will process the ${MESSAGE} part of the log message. You can also define the parser inline in the log path.

Declaration:

   parser parser_name {
        kv-parser(
            prefix()
        );
    };

Example: Using a key=value parser

In the following example, the source is a log message consisting of comma-separated key=value pairs, for example, a Postfix log message:

   Jun 20 12:05:12 mail.example.com <info> postfix/qmgr[35789]: EC2AC1947DA: from=<[email protected]>, size=807, nrcpt=1 (queue active)

The kv-parser inserts the “.kv.” prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.

   source s_kv {
        network(port(21514));
    };
    
    destination d_json {
        file("/tmp/test.json"
            template("$(format-json --scope dot-nv-pairs)\n"));
    };
    
    parser p_kv {
        kv-parser (prefix(".kv."));
    };
    
    log {
        source(s_kv);
        parser(p_kv);
        destination(d_json);
    };

You can also define the parser inline in the log path.

   source s_kv {
        network(port(21514));
    };
    
    destination d_json {
        file("/tmp/test.json"
            template("$(format-json --scope dot-nv-pairs)\n"));
    };
    
    log {
        source(s_kv);
        parser {
            kv-parser (prefix(".kv."));
        };
        destination(d_json);
    };

You can set the separator character between the key and the value to parse for example, key:value pairs, like MySQL logs:

   Mar  7 12:39:25 myhost MysqlClient[20824]: SYSTEM_USER:'oscar', MYSQL_USER:'my_oscar', CONNECTION_ID:23, DB_SERVER:'127.0.0.1', DB:'--', QUERY:'USE test;'

   parser p_mysql {
        kv-parser(value-separator(":") prefix(".mysql."));
    };

12.1 - Options of key=value parsers

The kv-parser has the following options.

extract-stray-words-into()

Description: Specifies the name-value pair where AxoSyslog stores any stray words that appear before or between the parsed key-value pairs (mainly when the pair-separator() option is also set). If multiple stray words appear in a message, then AxoSyslog stores them as a comma-separated list. Note that the prefix() option does not affect the name-value pair storing the stray words. Default value:N/A

Example: Extracting stray words in key-value pairs

For example, consider the following message:

   VSYS=public; Slot=5/1; protocol=17; source-ip=10.116.214.221; source-port=50989; destination-ip=172.16.236.16; destination-port=162;time=2016/02/18 16:00:07; interzone-emtn_s1_vpn-enodeb_om; inbound; policy=370;

This is a list of key-value pairs, where the value separator is = and the pair separator is ;. However, before the last key-value pair (policy=370), there are two stray words: interzone-emtn_s1_vpn-enodeb_om inbound. If you want to store or process these, specify a name-value pair to store them in the extract-stray-words-into() option, for example, extract-stray-words-into("my-stray-words"). The value of ${my-stray-words} for this message will be interzone-emtn_s1_vpn-enodeb_om, inbound

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, kv-parser() uses the .kv. prefix. To modify it, use the following format:

   parser {
        kv-parser(prefix("myprefix."));
    };

pair-separator()

Description: Specifies the character or string that separates the key-value pairs from each other. Default value: , .

For example, to parse key1=value1;key2=value2 pairs, use kv-parser(pair-separator(";")); .

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

value-separator()

Description: Specifies the character that separates the keys from the values. Default value: =.

For example, to parse key:value pairs, use kv-parser(value-separator(":"));.

13 - Linux audit parser

The Linux audit parser can parse the log messages of the Linux Audit subsystem (auditd). The AxoSyslog application can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The following is a sample log message of auditd:

   type=SYSCALL msg=audit(1441988805.991:239): arch=c000003e syscall=59 success=yes exit=0 a0=7fe49a6d0e98 a1=7fe49a6d0e40 a2=7fe49a6d0e80 a3=2 items=2 ppid=3652 pid=3660 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=5 comm="dumpe2fs" exe="/sbin/dumpe2fs" key=(null)
    type=EXECVE msg=audit(1441988805.991:239): argc=3 a0="dumpe2fs" a1="-h" a2="/dev/sda1"
    type=CWD msg=audit(1441988805.991:239):  cwd="/"
    type=PATH msg=audit(1441988805.991:239): item=0 name="/sbin/dumpe2fs" inode=137078 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
    type=PATH msg=audit(1441988805.991:239): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=5243184 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
    type=PROCTITLE msg=audit(1441988805.991:239): proctitle=64756D7065326673002D68002F6465762F73646131

Certain fields of the audit log can be encoded in hexadecimal format, for example, the arch field, or the a<number> fields in the previous example. The AxoSyslog application automatically decodes these fields (for example, the c000003e value becomes x86_64).

The AxoSyslog application extracts every field into name-value pairs. It automatically decodes the following fields:

• name

• proctitle

• path

• dir

• comm

• ocomm

• data

• old

• new

To parse the log messages of the Linux Audit subsystem, define a parser that has the linux-audit-parser() option. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message, use the template() option. You can also define the parser inline in the log path.

Declaration:

   parser parser_name {
        linux-audit-parser(
            prefix()
            template()
        );
    };

Example: Using the linux-audit-parser() parser

In the following example, the source is a log file created by auditd. Since the audit log format is not a syslog format, the syslog parser is disabled, so that AxoSyslog does not parse the message: flags(no-parse). The parser inserts “.auditd.” prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination, and the parser.

   source s_auditd {
        file(/var/log/audit/audit.log flags(no-parse));
    };
    
    destination d_json {
        file(
            "/tmp/test.json"
            template("$(format-json .auditd.*)\n")
        );
    };
    
    parser p_auditd {
        linux-audit-parser (prefix(".auditd."));
    };
    
    log {
        source(s_auditd);
        parser(p_auditd);
        destination(d_json);
    };

You can also define the parser inline in the log path.

   source s_auditd {
        file(/var/log/audit/audit.log);
    };
    
    destination d_json {
        file(
            "/tmp/test.json"
            template("$(format-json .auditd.*)\n")
        );
    };
    
    log {
        source(s_auditd);
        parser {
            linux-audit-parser (prefix(".auditd."));
        };
        destination(d_json);
    };

13.1 - Options of linux-audit-parser() parsers

The linux-audit-parser() has the following options.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, linux-audit-parser() uses the .auditd. prefix. To modify it, use the following format:

   parser {
        linux-audit-parser(prefix("myprefix."));
    };

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

14 - MariaDB parser

The MariaDB parser can parse the log messages of the MariaDB Audit Plugin. The parser supports the syslog output typess’ format. Available in version 3.37 and later.

Declaration:

   @version: 4.5
    @include "scl.conf"
    log {
        source { system(); };
        parser { mariadb-audit-parser(); };
        destination { ... };
    };

The mariadb-audit is a reusable configuration snippet configured to parse MariaDB Audit Plugin messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, mariadb-audit uses the .mariadb. prefix. To modify it, use the following format:

   parser {
        mariadb-audit-parser(prefix("myprefix."));
    };

15 - metrics-probe

Available in AxoSyslog version 4.1.1 and newer.

metrics-probe() is a special parser that counts the messages that pass through the log path, and creates labeled stats counters based on the fields of the passing messages.

You can configure the name of the keys and the labels. Note that the keys are automatically prefixes with the syslogng_ string. You can use templates in the values of the labels.

The minimal configuration creates counters with the key syslogng_classified_events_total and labels app, host, program and source. For example:

parser p_metrics_probe {
  metrics-probe();

  # Same as:
  #
  # metrics-probe(
  #   key("classified_events_total")
  #   labels(
  #     "app" => "${APP}"
  #     "host" => "${HOST}"
  #     "program" => "${PROGRAM}"
  #     "source" => "${SOURCE}"
  #   )
  # );
};

This configuration results in counters like:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

You can query the metrics by running the following command:

syslog-ng-ctl stats prometheus

For example, the following metrics-probe() parser creates a counter called syslogng_custom_key that counts messages that have their custom_label_name_1 field set to foobar, and for these messages it creates separate counters based on the value of the custom_label_name_2 field.

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

This configuration results in counters like:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

Starting with AxoSyslog 4.4, you can create dynamic labels as well.

Options

increment()

Available in AxoSyslog version 4.2 and newer.

Sets a template, which resolves to a number that defines the increment of the counter. The following example defines a counter called syslogng_input_event_bytes_total, and increases its value with the size of the incoming message (in bytes).

metrics-probe(
    key("input_event_bytes_total")
    labels(
        "cluster" => "`cluster-name`"
        "driver" => "kubernetes"
        "id" => "${SOURCE}"
        "namespace" => "${`prefix`namespace_name}"
        "pod" => "${`prefix`pod_name}"
    )
    increment("${RAWMSG_SIZE}")
);

key()

The name of the counter to create. Note that the value of this option is always prefixed with syslogng_, so for example key("my-custom-key") becomes syslogng_my-custom-key.

labels()

The labels used to create separate counters, based on the fields of the messages processed by metrics-probe(). Use the following format:

labels(
    "name-of-the-label-in-the-output" => "field-of-the-message"
)

Default value:

labels(
    "app" => "${APP}"
    "host" => "${HOST}"
    "program" => "${PROGRAM}"
    "source" => "${SOURCE}"
)

This results in counters like:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3

Dynamic labels

Available in AxoSyslog 4.4 and newer.

Dynamic labelling allows you to use every available value-pairs() options in the labels, for example, key(), rekey(), pair(), or scope().

For example:

metrics-probe(
  key("foo")
  labels(
    "static-label" => "bar"
    key(".my_prefix.*" rekey(shift-levels(1)))
  )
);

syslogng_foo{static_label="bar",my_prefix_baz="anotherlabel",my_prefix_foo="bar",my_prefix_nested_axo="flow"} 4

level()

Available in AxoSyslog version 4.2 and newer.

Sets the stats level of the generated metrics.

Note: Drivers configured with internal(yes) register their metrics on level 3. That way if you are creating an SCL, you can disable the built-in metrics of the driver, and create metrics manually using metrics-probe().

16 - Netskope parser

The Netskope parser can parse Netskope log messages. These messages do not completely comply with the syslog RFCs, making them difficult to parse. The netskope-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:

   <PRI>{JSON-formatted-log-message}

For example:

   <134>{"count": 1, "supporting_data": {"data_values": ["x.x.x.x", "[email protected]"], "data_type": "user"}, "organization_unit": "domain/domain/Domain Users/Enterprise Users", "severity_level": 2, "category": null, "timestamp": 1547421943, "_insertion_epoch_timestamp": 1547421943, "ccl": "unknown", "user": "[email protected]", "audit_log_event": "Login Successful", "ur_normalized": "[email protected]", "_id": "936289", "type": "admin_audit_logs", "appcategory": null}

If you find a message that the netskope-parser() cannot properly parse, contact us, so we can improve the parser.

The AxoSyslog application sets the ${PROGRAM} field to Netskope.

By default, the Netskope-specific fields are extracted into name-value pairs prefixed with .netskope. For example, the organization_unit in the previous message becomes ${.netskope.organization_unit}. You can change the prefix using the prefix option of the parser.

Declaration:

   @version: 4.5.0
    @include "scl.conf"
    log {
        source { network(flags(no-parse)); };
        parser { netskope-parser(); };
        destination { ... };
    };

Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.

The netskope-parser() is actually a reusable configuration snippet configured to parse Netskope messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.