JSON parser

• 1: Options of JSON parsers

   parser parser_name {
        json-parser(
            marker()
            prefix()
        );
    };

   source s_json {
        network(
            port(21514
            flags(no-parse)
        );
    };
    
    destination d_json {
        file(
            "/tmp/test.json"
            template("$(format-json --scope dot-nv-pairs)\n")
        );
    };
    
    parser p_json {
        json-parser (prefix(".json."));
    };
    
    log {
        source(s_json);
        parser(p_json);
        destination(d_json);
    };

   source s_json {
        network(
            port(21514)
            flags(no-parse)
        );
    };
    
    destination d_json {
        file(
            "/tmp/test.json"
            template("$(format-json --scope dot-nv-pairs)\n")
        );
    };
    
    log {
        source(s_json);
        parser {
            json-parser (prefix(".json."));
        };
        destination(d_json);
    };

1 - Options of JSON parsers

The JSON parser has the following options.

extract-prefix()

Description: Extract only the specified subtree from the JSON message. Use the dot-notation to specify the subtree. The rest of the message will be ignored. For example, assuming that the incoming object is named msg, the json-parser(extract-prefix("foo.bar[5]")); parser is equivalent to the msg.foo.bar[5] javascript code. Note that the resulting expression must be a JSON object in order to extract its members into name-value pairs.

This feature also works when the top-level object is an array, because you can use an array index at the first indirection level, for example: json-parser(extract-prefix("[5]")), which is equivalent to msg[5].

In addition to alphanumeric characters, the key of the JSON object can contain the following characters: \!"#$%&'()\*+,-/:;<=>?@\\^_{|}~`

It cannot contain the following characters: .[]

Example: Convert logstash eventlog format v0 to v1

The following parser converts messages in the logstash eventlog v0 format to the v1 format.

   parser p_jsoneventv0 {
        channel {
            parser {
                json-parser(extract-prefix("@fields"));
            };
            parser {
                json-parser(prefix(".json."));
            };
            rewrite {
                set("1" value("@version"));
                set("${.json.@timestamp}" value("@timestamp"));
                set("${.json.@message}" value("message"));
            };
        };
    };

key-delimiter()

Description: The delimiter character to use when parsing flattened keys. Supports Only single characters.

marker

Description: Use a marker in case of mixed log messages, to identify JSON encoded messages for the parser.

Some logging implementations require a marker to be set before the JSON payload. The JSON parser is able to find these markers and parse the message only if it is present.

Example: Using the marker option in JSON parser

This json parser parses log messages which use the “@cee:” marker in front of the json payload. It inserts “.cee.” in front of the name of name-value pairs, so later on it is easier to find name-value pairs that were parsed using this parser. (For details on selecting name-value pairs, see value-pairs().)

   parser {
            json-parser(
                marker("@cee:")
                prefix(".cee.")
            );
        };

prefix()

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:

• To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

• To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.

• If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

This parser does not have a default prefix. To configure a custom prefix, use the following format:

   parser {
        json-parser(prefix("myprefix."));
    };

template()

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).