Options of CSV parsers
The parse_csv
FilterX function has the following options.
columns
Synopsis: | columns=["1st","2nd","3rd"] |
Default value: | N/A |
Description: Specifies the names of the columns, and correspondingly the keys in the resulting JSON array.
- If the
columns
option is set,parse_csv
returns a dictionary with the column names (as keys) and the parsed values. - If the
columns
option isn’t set,parse_csv
returns a list.
delimiter
Synopsis: | delimiter="<string-with-delimiter-characters>" |
Default value: | , |
Description: The delimiter parameter contains the characters that separate the columns in the input string. If you specify multiple characters, every character will be treated as a delimiter. Note that the delimiters aren’t included in the column values. For example:
- To separate the text at every hyphen (-) and colon (:) character, use
delimiter="-:"
. - To separate the columns along the tabulator (tab character), specify
delimiter="\\t"
. - To use strings instead of characters as delimiters, see
string_delimiters
.
Multiple delimiters
If you use more than one delimiter, note the following points:
- AxoSyslog will split the message at the nearest possible delimiter. The order of the delimiters in the configuration file does not matter.
- You can use both string delimiters and character delimiters in a parser.
- The string delimiters may include characters that are also used as character delimiters.
- If a string delimiter and a character delimiter both match at the same position of the input, AxoSyslog uses the string delimiter.
dialect
Synopsis: | dialect="<dialect-name>" |
Default value: | escape-none |
Description: Specifies how to handle escaping in the input strings.
The following values are available.
escape-backslash
: The parsed message uses the backslash (\\
) character to escape quote characters.escape-backslash-with-sequences
: The parsed message uses""
as an escape character but also supports C-style escape sequences, like\n
or\r
. Available in AxoSyslog version 4.0 and later.escape-double-char
: The parsed message repeats the quote character when the quote character is used literally. For example, to escape a comma (,
), the message contains two commas (,,
).escape-none
: The parsed message does not use any escaping for using the quote character literally.
greedy
Synopsis: | greedy=true |
Default value: | false |
If the greedy
option is enabled, AxoSyslog adds the remaining part of the message to the last column, ignoring any delimiters that may appear in this part of the message. You can use this option to process messages where the number of columns varies from message to message.
For example, you receive the following comma-separated message: example 1, example2, example3
, and you segment it with the following parser:
my-parsed-values = parse_csv(${MESSAGE}, columns=["COLUMN1", "COLUMN2", "COLUMN3"], delimiter=",");
The COLUMN1
, COLUMN2
, and COLUMN3
variables will contain the strings example1
, example2
, and example3
, respectively. If the message looks like example 1, example2, example3, some more information
, then any text appearing after the third comma (that is, some more information
) is not parsed, and thus possibly lost if you use only the parsed columns to reconstruct the message (for example, if you send the columns to different columns of a database table).
Using the greedy=true
flag will assign the remainder of the message to the last column, so that the COLUMN1
, COLUMN2
, and COLUMN3
variables will contain the strings example1
, example2
, and example3, some more information
.
my-parsed-values = parse_csv(${MESSAGE}, columns=["COLUMN1", "COLUMN2", "COLUMN3"], delimiters=[","], greedy=true);
strip_whitespace
Synopsis: | strip_whitespace=true |
Default value: | false |
Description: Remove leading and trailing whitespaces from all columns. The strip_whitespaces
option is an alias for strip_whitespace
.
string_delimiters
Synopsis: | string_delimiters=json_array(["first-string","2nd-string"]) |
Description: In case you have to use a string as a delimiter, list your string delimiters as a JSON array in the string_delimiters=["<delimiter_string1>", "<delimiter_string2>", ...]
option.
By default, the parse_csv
FilterX function uses the comma as a delimiter. If you want to use only strings as delimiters, you have to disable the default space delimiter, for example: delimiter="", string_delimiters=["<delimiter_string>"])
Otherwise, AxoSyslog will use the string delimiters in addition to the default character delimiter, so for example, string_delimiters=["=="]
is actually equivalent to delimiters=",", string_delimiters=["=="]
, and not delimiters="", string_delimiters=["=="]
Multiple delimiters
If you use more than one delimiter, note the following points:
- AxoSyslog will split the message at the nearest possible delimiter. The order of the delimiters in the configuration file does not matter.
- You can use both string delimiters and character delimiters in a parser.
- The string delimiters may include characters that are also used as character delimiters.
- If a string delimiter and a character delimiter both match at the same position of the input, AxoSyslog uses the string delimiter.