This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Parsing key=value pairs

The AxoSyslog application can separate a message consisting of whitespace or comma-separated key=value pairs (for example, Postfix log messages) into name-value pairs. You can also specify other separator character instead of the equal sign, for example, colon (:) to parse MySQL log messages. The AxoSyslog application automatically trims any leading or trailing whitespace characters from the keys and values, and also parses values that contain unquoted whitespace. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs.

You can refer to the separated parts of the message using the key of the value as a macro. For example, if the message contains KEY1=value1,KEY2=value2, you can refer to the values as ${KEY1} and ${KEY2}.

To parse key=value pairs, define a parser that has the kv-parser() option. Defining the prefix is optional. By default, the parser will process the ${MESSAGE} part of the log message. You can also define the parser inline in the log path.

Declaration:

   parser parser_name {
        kv-parser(
            prefix()
        );
    };

Example: Using a key=value parser

In the following example, the source is a log message consisting of comma-separated key=value pairs, for example, a Postfix log message:

   Jun 20 12:05:12 mail.example.com <info> postfix/qmgr[35789]: EC2AC1947DA: from=<[email protected]>, size=807, nrcpt=1 (queue active)

The kv-parser inserts the “.kv.” prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.

   source s_kv {
        network(port(21514));
    };
    
    destination d_json {
        file("/tmp/test.json"
            template("$(format-json --scope dot-nv-pairs)\n"));
    };
    
    parser p_kv {
        kv-parser (prefix(".kv."));
    };
    
    log {
        source(s_kv);
        parser(p_kv);
        destination(d_json);
    };

You can also define the parser inline in the log path.

   source s_kv {
        network(port(21514));
    };
    
    destination d_json {
        file("/tmp/test.json"
            template("$(format-json --scope dot-nv-pairs)\n"));
    };
    
    log {
        source(s_kv);
        parser {
            kv-parser (prefix(".kv."));
        };
        destination(d_json);
    };

You can set the separator character between the key and the value to parse for example, key:value pairs, like MySQL logs:

   Mar  7 12:39:25 myhost MysqlClient[20824]: SYSTEM_USER:'oscar', MYSQL_USER:'my_oscar', CONNECTION_ID:23, DB_SERVER:'127.0.0.1', DB:'--', QUERY:'USE test;'
   parser p_mysql {
        kv-parser(value-separator(":") prefix(".mysql."));
    };

1 - Options of key=value parsers

The kv-parser has the following options.

extract-stray-words-into()

Synopsis: extract-stray-words-into(")

Description: Specifies the name-value pair where AxoSyslog stores any stray words that appear before or between the parsed key-value pairs (mainly when the pair-separator() option is also set). If multiple stray words appear in a message, then AxoSyslog stores them as a comma-separated list. Note that the prefix() option does not affect the name-value pair storing the stray words. Default value:N/A

Example: Extracting stray words in key-value pairs

For example, consider the following message:

   VSYS=public; Slot=5/1; protocol=17; source-ip=10.116.214.221; source-port=50989; destination-ip=172.16.236.16; destination-port=162;time=2016/02/18 16:00:07; interzone-emtn_s1_vpn-enodeb_om; inbound; policy=370;

This is a list of key-value pairs, where the value separator is = and the pair separator is ;. However, before the last key-value pair (policy=370), there are two stray words: interzone-emtn_s1_vpn-enodeb_om inbound. If you want to store or process these, specify a name-value pair to store them in the extract-stray-words-into() option, for example, extract-stray-words-into("my-stray-words"). The value of ${my-stray-words} for this message will be interzone-emtn_s1_vpn-enodeb_om, inbound

prefix()

Synopsis: prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

  • To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

  • To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

  • If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

By default, kv-parser() uses the .kv. prefix. To modify it, use the following format:

   parser {
        kv-parser(prefix("myprefix."));
    };

pair-separator()

Synopsis: pair-separator(")

Description: Specifies the character or string that separates the key-value pairs from each other. Default value: , .

For example, to parse key1=value1;key2=value2 pairs, use kv-parser(pair-separator(";")); .

template()

Synopsis: template("${<macroname>}")

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

value-separator()

Synopsis: value-separator("")

Description: Specifies the character that separates the keys from the values. Default value: =.

For example, to parse key:value pairs, use kv-parser(value-separator(":"));.