FilterX (developed by Axoflow) is a replacement for syslog-ng filters, parsers, and rewrite rules. It has its own syntax, allowing you to filter, parse, manipulate, and rewrite variables and complex data structures, and also compare them with various operators.
FilterX is a consistent and comprehensive reimplementation of several core features with improved performance, proper typing support, and the ability to handle multi-level typed objects.
FilterX helps you to route, parse, and modify your logs: a message passes through the FilterX block in a log path only if all the FilterX statements evaluate to true for the particular message. If a log statement includes multiple FilterX blocks, the messages are sent to the destinations only if they pass all FilterX blocks of the log path. For example, you can select only the messages originating from a particular host, or create complex filters using operators, functions, and logical expressions.
FilterX blocks consist of a list of FilterX statements, each statement evaluates either to truthy or falsy. If a message matches all FilterX statements, it passes through the FilterX block to the next element of the log path, for example, the destination.
Truthy values are:
Complex values (for example, a datetime object),
non-empty lists and objects,
non-empty strings,
non-zero numbers,
the true boolean object.
Falsy values are:
empty strings,
the false value,
the 0 value,
null,
Statements that result in an error (for example, if a comparison cannot be evaluated because of type error, or a field or a dictionary referenced in the statement doesn’t exist or is unset) are also treated as falsy.
Define a filterx block
You can define filterx blocks inline in your log statements. (If you want to reuse filterx blocks, Reuse FilterX blocks.)
For example, the following FilterX statement selects the messages that contain the word deny and come from the host example.
You can use filterx blocks together with other blocks in a log path, for example, use a parser before/after the filterx block if needed.
FilterX statements
A FilterX block contains one or more FilterX statements. The order of the statements is important, as they are processed sequentially. If any of the statements is falsy (or results in an error), AxoSyslog drops the message from that log path.
FilterX statements can be one of the following:
A comparison, for example, ${HOST} == "my-host";. This statement is true only for messages where the value of the ${HOST} field is my-host. Such simple comparison statements can be the equivalents of traditional filter functions.
A value assignment for a name-value pair or a local variable, for example, ${my-field} = "bar";. The left-side variable automatically gets the type of the right-hand expression. Assigning the false value to a variable (${my-field} = false;) is a valid statement that doesn’t automatically cause the FilterX block to return as false.
Existence of a variable of field. For example, the ${HOST}; expression is true only if the ${HOST} macro exists and isn’t empty.
A conditional statement ( if (expr) { ... } elif (expr) {} else { ... };) which allows you to evaluate complex decision trees. Starting with version 4.10, you can also use switch-case expressions.
A declaration of a pipeline variable, for example, declare my_pipeline_variable = "something";.
A FilterX action. This can be one of the following:
drop;: Intentionally drop the message. This means that the message was successfully processed, but discarded. Processing the dropped message stops at the drop statement, subsequent sections or other branches of the FilterX block won’t process the message. For example, you can use this to discard unneeded messages, like debug logs. Available in AxoSyslog 4.9 and later.
done;: Return truthy and don’t execute the rest of the FilterX block, returns with true. This is an early return that you can use to avoid unnecessary processing, for example, when the message matches an early classification in the block. Available in AxoSyslog 4.9 and later.
Note
The true; and false; literals are also valid as statements. They can be useful in complex conditional (if/elif/else) statements.
A name-value pair or a variable in itself is also a statement. For example, ${HOST};. If the name-value pair or variable is empty or doesn’t exist, the statement is considered falsy.
When you assign the value of a variable using another variable (for example, ${MESSAGE} = ${HOST};), AxoSyslog copies the current value of the ${HOST} variable. If a statement later changes the value of the ${HOST} field, the ${MESSAGE} field won’t change. For example:
Terminal window
filterx {${HOST}="first-hostname";${MESSAGE}=${HOST};# The value of ${MESSAGE} is first-hostname${HOST}="second-hostname";# The value of ${MESSAGE} is still first-hostname};
The same is true for complex objects, like JSON, for example:
Each FilterX block can access data from the following elements.
Macros and name-value pairs of the message being processed (for example, $PROGRAM). The names of macros and name-value pairs begin with the $ character. If you define a new variable in a FilterX block and its name begins with the $ character, it’s automatically added to the name-value pairs of the message.
Note
Using curly braces around macro names is not mandatory, and the "$MESSAGE" and "${MESSAGE}" formats are equivalent. If the name contains only alphanumeric characters and the underscore character, you don’t need the curly braces. If it contains any other characters (like a hyphen (-) or a dot (.)), you need to add the curly braces, therefore it’s best to always use curly braces.
Names are case-sensitive, so "$message" and "$MESSAGE" are not the same.
Local variables. These have a name that doesn’t start with a $ character, for example, my_local_variable. Local variables are available only in the FilterX block they’re defined.
Pipeline variables. These are similar to local variables, but must be declared before first use, for example, declare my_pipeline_variable=5;
Pipeline variables are available in the current and all subsequent FilterX block. They’re global in the sense that you can access them from multiple FilterX blocks, but note that they’re still attached to the particular message that is processed, so the values of pipeline variables aren’t preserved between messages.
If you don’t need to pass the variable to another FilterX block, use local variables, as pipeline variables have a slight performance overhead.
Note
If you want to pass data between two FilterX blocks of a log statement, use pipeline variables. That has better performance than name-value pairs.
You can also assign the value of other name-value pairs, for example:
Terminal window
filterx {${MESSAGE}=${HOST};};
When processing RFC5424-formatted (IETF-syslog) messages, you can modify the SDATA part of the message as well. The following example sets the sequenceId:
Terminal window
filterx {${.SDATA.meta.sequenceId}= 55555;};
Note
When assigning values to name-value pairs, you cannot modify hard macros.
Template functions
You can use the traditional template functions of AxoSyslog to access and format name-value pairs. For that you must enclose the template function expression between double-quotes, for example:
Terminal window
${MESSAGE}="$(format-json --subkeys values.)";
However, note that template functions cannot access the local and pipeline variables created in FilterX blocks.
Delete values
To delete a value without deleting the object itself (for example, name-value pair), use the null value, for example:
Terminal window
${MY-NV-PAIR-KEY}= null;
To delete the name-value pair (or a key from an object), use the unset function:
The plus operator (+) adds two arguments, if possible. (For example, you can’t add two datetime values.)
You can use it to add two numbers (two integers, two double values). If you add a double to an integer, the result is a double.
Adding two strings concatenates the strings. Note that if you want to have spaces between the added elements, you have to add them manually, like in Python, for example:
Terminal window
${MESSAGE}=${HOST} + " first part of the message," + " second part of the message" + "\n";
Adding two lists merges the lists. Available in AxoSyslog 4.9 and later.
Adding two dicts updates the dict with the values of the second operand. For example:
Terminal window
x={"key1": "value1", "key2": "value1"};y={"key3": "value1", "key2": "value2"};${MESSAGE}= x + y;# ${MESSAGE} value is {"key1": "value1", "key3": "value1", "key2": "value2"};
The list and dict types are similar to their Python counterparts. FilterX uses JSON to represent generic dictionary and list types, but you can create other, specific dictionary and list types as well (currently for OTEL, for example, otel_kvlist, or otel_array). All supported dictionary and list types are compatible with each other, and you can convert them to and from each other, copy values between them (retaining the type), and so on.
For example:
Terminal window
my_list=[];# Creates an empty list (which defaults to a JSON list)my_array={};# Creates an empty dictionary (which defaults to a JSON object)my_list2= json_array();# Creates an empty JSON listmy_array2= json();# Creates an empty JSON object.
You can add elements to lists and dictionaries like this:
Terminal window
list= json_array();# Create an empty JSON list#list = otel_array(); # Create an OTEL listlist+=["first_element"];# Append entries to the listlist+=["second_element"];list+=["third_element"];${MESSAGE}= list;
You can also create the list and assign values in a single step:
You can refer to the elements using an index (starting with 0):
Terminal window
list= json_array();# Create an empty JSON listlist[0]="first_element";# Append entries to the listlist[1]="second_element";list[2]="third_element";${MESSAGE}= list;
In all three cases, the value of ${MESSAGE} is the same JSON array: ["first_element", "second_element", "third_element"].
You can define JSON objects using the json() type, for example:
Within a FilterX block, you can access the fields of complex data types by using indexes and the dot notation, for example:
dot notation: js.key
indexing: js["key"]
or mixed mode if needed: js.list[1]
When referring to the field of a name-value pair (which begins with the $ character), place the dot or the square bracket outside the curly bracket surrounding the name of the name-value pair, for example: ${MY-LIST}[2] or ${MY-OBJECT}.mykey. If the name of the key contains characters that are not permitted in FilterX variable names, for example, a hyphen (-), use the bracketed syntax and enclose the key in double quotes: ${MY-LIST}["my-key-name"].
You can add two lists or two dicts using the Plus operator.
Tuples
Available in AxoSyslog 4.27 and later.
A tuple is a read-only, list-like data type similar to a Python tuple. You can initialize a tuple only once, then it remains read-only until the end of its lifecycle.
Tuple examples:
Terminal window
t=();# empty tuplet=("foo",);# singletont=(1,2,3);# a tuple of 3 elementsd={'foo':'foovalue','bar':'barvalue'};t= tuple([1,2,3,d]);# a tuple with the value (1,2,3,{"foo":"foovalue","bar":"barvalue"})
List membership
Available in AxoSyslog 4.12 and later.
You can check whether a value is in a list using the in and not in membership operators. If you’re checking the value of a variable that can have another type than the elements of a list, explicitly convert the value to the correct type (for example, string) before the comparison.
Terminal window
my_array= json_array(['hostname_one', 'hostname_two', 'hostname_three']);if(${HOST} in my_array){# ...}
Starting with AxoSyslog 4.22, you can also use the operators to check membership in dict keys. For example:
The in operator can be used to replicate the functionality of the in-list() filter function in FilterX. If you want to populate the list from a file, use the cache_json_file() FilterX function.
endswith: Checks if a string ends with the specified value.
flatten: Flattens the nested elements of an object.
fix_timezone: Corrects the timezone of a message if it was parsed incorrectly for some reason, or if the client didn’t include any timezone information in the message.
format_cef: Formats a dictionary into Common Event Format (CEF).
format_csv: Formats a dictionary or a list into a comma-separated string.
format_isodate: Formats a date as ISODATE: %Y-%m-%dT%H:%M:%S%z
The following list shows you some common tasks that you can solve with FilterX:
To set message fields (like macros or SDATA fields) or replace message parts: you can assign values to change parts of the message, or use one of the FilterX functions to rewrite existing values.
use value comparison in the FilterX block to select the appropriate messages. For example, to rewrite only messages of the NGINX application, you can:
Terminal window
${PROGRAM}=="nginx";# <your rewrite expression>
Create an iptables parser
The following example shows you how to reimplement the iptables parser in a FilterX block. The following is a sample iptables log message (with line-breaks added for readability):
Terminal window
Dec 08 12:00:00 hostname.example kernel: custom-prefix:IN=eth0 OUT=MAC=11:22:33:44:55:66:aa:bb:cc:dd:ee:ff:08:00 SRC=192.0.2.2 DST=192.168.0.1 LEN=40TOS=0x00
PREC=0x00 TTL=232ID=12345PROTO=TCP SPT=54321DPT=22WINDOW=1023RES=0x00 SYN URGP=0
This is a normal RFC3164-formatted message logged by the kernel (where iptables logging messages originate from), and contains space-separated key-value pairs.
First, create some filter statements to select iptables messages only:
Terminal window
block filterx parse_iptables(){${FACILITY}=="kern";# Filter on the kernel facility${PROGRAM}=="kernel";# Sender application is the kernel${MESSAGE}=~ "PROTO=";# The PROTO key appears in all iptables messages}
To make the parsed data available under macros beginning with ${.iptables}, like in the case of the original iptables-parser(), create the ${.iptables} JSON object.
Terminal window
block filterx parse_iptables(){${FACILITY}=="kern";# Filter on the kernel facility${PROGRAM}=="kernel";# Sender application is the kernel${MESSAGE}=~ "PROTO=";# The PROTO key appears in all iptables messages${.iptables}= json();# Create an empty JSON object}
Add a key=value parser to parse the content of the messages into the ${.iptables} JSON object. The key=value pairs are space-separated, while equal signs (=) separates the values from the keys.
Terminal window
block filterx parse_iptables(){${FACILITY}=="kern";# Filter on the kernel facility${PROGRAM}=="kernel";# Sender application is the kernel${MESSAGE}=~ "PROTO=";# The PROTO key appears in all iptables messages${.iptables}= json();# Create an empty JSON object${.iptables}= parse_kv(${MESSAGE}, value_separator="=", pair_separator=" ");}
If you’re modifying messages using FilterX (for example, you extract a value from the message and add it to another field of the message), note the following points:
Macros and name-value pairs (variables with names beginning with the $ character) are included in the outgoing message in case the template of the destination includes them. For example, if you change the value of the ${MESSAGE} macro, it’s automatically sent to the destination if the destination template includes this macro.
Local and pipeline variables are not included in the message, you must assign their value to a macro or name-value pair that’s included in the destination template to send them to the destination.
When sending data to opentelemetry() destinations, if you’re modifying messages received via the opentelemetry() source, then you must explicitly update the original (raw) data structures in your FilterX block, otherwise the changes won’t be included in the outgoing message. For details, see Modify incoming OTEL.
1 - Boolean operators in FilterX
When a log statement includes multiple filter statements, AxoSyslog sends a message to the destination only if all filters are true for the message. In other words, the filters are connected by logical AND operators. In the following example, no message arrives to the destination, because the filters are mutually exclusive (the hostname of a client cannot be example1 and example2 at the same time):
When checking for equality (==), sometimes it’s also important to check that the two operands have the same type. For that purpose, you can use the === (strict equality) operator.
However, to select the messages that weren’t sent by host example1 or example2, you have to use the and operator (that’s how boolean logic works, see De Morgan’s laws for details):
Terminal window
filterx { not (${HOST}=="example1") and not (${HOST}=="example2");};
Alternatively, you can use parentheses and the or operator to avoid this confusion:
Terminal window
filterx { not ((${HOST}=="example1") or (${HOST}=="example2"));};
The following filter statement selects the messages that contain the word deny and come from the host example.
Note
FilterX blocks are often used together with log path flags. For details, see Log path flags.
2 - Comparing values in FilterX
In AxoSyslog you can compare macro values, templates, and variables as numerical and string values. String comparison is alphabetical: it determines if a string is alphabetically greater than or equal to another string. For details on macros and templates, see Customize message format using macros and templates.
Use the following syntax to compare macro values or templates.
You can use mathematical symbols as operators (like ==, !=, >=), and based on the type of the arguments AxoSyslog automatically determines how to compare them. The logic behind this is similar to JavaScript:
If both sides of the comparisons are strings, then the comparison is string.
Note: Comparing strings is case sensitive. For case insensitive comparison, use the strcasecmp() function.
If one of the arguments is numeric, then the comparison is numeric.
Literal numbers (numbers not enclosed in quotes) are numeric.
You can explicitly type-cast an argument into a number.
The bytes, json, and protobuf types are always compared as strings.
Currently you can’t compare dictionaries and lists.
For example:
if (${.apache.httpversion} == 1.0)
The right side of the == operator is 1.0, which is a floating point literal (double), so the comparison is numeric.
if (double(${.apache.httpversion}) == "1.0")
The left side is explicitly type cast into double, the right side is string (because of the quotes), so the comparison is numeric.
if (${.apache.request} == "/wp-admin/login.php")
The left side is not type-cast, the right side is a string, so the comparison is string, and case sensitive. The case insensitive equivalent is: if (strcasecmp(${.apache.request}, "/wp-admin/login.php") == 0)
Note
You can use string operators if you want to, they are still available for backwards compatibility.
Example: Compare macro values
The following expression selects log messages that contain a PID (that is, the ${PID} macro is not empty):
Terminal window
filterx {${PID};};
(It is equivalent to using the isset() function: isset(${PID});).
The following expression selects log messages where the priority level is not emerg.
Terminal window
filterx {${LEVEL} !="emerg";};
The following example selects messages with priority level higher than 5.
Terminal window
filterx {${LEVEL_NUM} > 5;};
Make sure to:
Enclose literal strings and templates in double-quotes. For macros and variables do not use quotes.
Use the $ character before macros.
Note that you can use:
type casting anywhere where you can use templates to apply a type to the result of the template expansion.
any macro in the expression, including user-defined macros from parsers and classifications.
boolean operators to combine comparison expressions.
Compare the type (strict equality)
To compare the values of operands and verify that they have the same type, use the === (strict equality) operator. The following example defines a string variable with the value “5” as string and uses it in different comparisons:
Terminal window
mystring="5";# Type is stringmystring=== 5;# false, because the right-side is an integermystring==="5";# true};
To compare only the types of variables and macros, you can use the istype function.
Strict inequality operator
Compares the values of operands and returns true if they are different. Also returns true if the value of the operands are the same, but their type is different. For example:
Terminal window
"example" !=="example";# False, because they are the same and both are strings"1" !== 1;# True, because one is a string and the other an integer
Comparison operators
The following numerical and string comparison operators are available.
Numerical or string operator
String operator
Meaning
==
eq
Equals
!=
ne
Not equal to
>
gt
Greater than
<
lt
Less than
>=
ge
Greater than or equal
=<
le
Less than or equal
===
Equals and has the same type
!==
Not equal to or has different type
3 - Conditional statements
A conditional statement ( if (expr) { ... } elif (expr) {} else { ... };) allows you to evaluate complex decision trees. For example:
Starting with version 4.10, you can also use switch-case expressions. Switch-case expressions allow you to better organize the code instead of using multiple if, elif, else blocks. Using switch-case expressions also improves performance.
Cases with literal string targets are stored in a map, and the lookup is started with them.
Case targets can contain any expressions, and they are evaluated in order.
Literal string and default target duplications are checked and will cause init failure. Non-literal expression targets are not checked, and only the first matching case will be executed.
4 - String search in FilterX
Available in AxoSyslog 4.9 and later.
You can check if a string contains a specified string using the includes FilterX function. The startswith and endswith functions check the beginning and ending of the strings, respectively. For example, the following expression checks if the message ($MESSAGE) begins with the %ASA- string:
Terminal window
startswith($MESSAGE, '%ASA-')
By default, matches are case sensitive. For case insensitive matches, use the ignorecase=true option:
Terminal window
startswith($MESSAGE, '%ASA-', ignorecase=true)
All three functions (includes, startswith, and endswith) can take a list with multiple search strings and return true if any of them match. This is equivalent with using combining the individual searches with logical OR operators. For example:
Terminal window
${MESSAGE}="%ASA-5-111010: User ''john'', running ''CLI'' from IP 0.0.0.0, executed ''dir disk0:/dap.xml"includes($MESSAGE, ['%ASA-','john','CLI'])includes($MESSAGE, ['%ASA-','john','CLI'])includes($MESSAGE, '%ASA-') or includes($MESSAGE, 'john') or includes($MESSAGE, 'CLI')
Starting with AxoSyslog version 4.19, the includes function has a limit option to truncate the search to the first <limit> character of the string. The following example searches for the string john only in the first 40 characters of the $MESSAGE:
Terminal window
${MESSAGE}="%ASA-5-111010: User ''john'', running ''CLI'' from IP 0.0.0.0, executed ''dir disk0:/dap.xml"includes($MESSAGE, 'john', limit=40)
The first argument is the input message. Optionally, you can set the pair_separator and value_separator arguments to override their default values.
The value_separator must be a single-character string. The pair_separator can be a regular string.
Starting with version 4.13, AxoSyslog parses fields from extensions to the same level as regular fields. In earlier versions, extensions were grouped under the extensions key. To keep using the extensions key, set separate_extensions=true.
The parsed JSON object has the following fields:
cef_version
device_vendor
device_product
device_version
device_event_class_id
event_name
agent_severity
extensions
Note
The name of some fields changed in the parsed object in version 4.16 for clarity, and to avoid name collisions with fields in the extensions:
version -> cef_version
name -> event_name
Example
The following is a CEF-formatted message including mandatory and custom (extension) fields:
The parse_cef FilterX function has the following options.
pair_separator
Specifies the character or string that separates the key-value pairs in the extensions. Default value: (space).
separate_extensions
Available in AxoSyslog 4.13 and later.
Starting with version 4.13, AxoSyslog parses fields from extensions to the same level as regular fields. In earlier versions, extensions were grouped under the extensions key. To keep using the extensions key, set separate_extensions=true.
Default value: false
value_separator
Specifies the character that separates the keys from the values in the extensions. Default value: =.
5.2 - Comma-separated values
The parse_csv FilterX function can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) along delimiter characters or strings into lists, or key-value pairs within dictionaries, using the csv (comma-separated-values) parser.
If the columns option is set, parse_csv returns a dictionary with the column names (as keys) and the parsed values. If the columns option isn’t set, parse_csv returns a list.
The following example separates hostnames like example-1 and example-2 into two parts.
Terminal window
block filterx p_hostname_segmentation(){cols= json_array(["NAME","ID"]);HOSTNAME= parse_csv(${HOST}, delimiter="-", columns=cols);# HOSTNAME is a json object containing parts of the hostname# For example, for example-1 it contains:# {"NAME":"example","ID":"1"}# Set the important elements as name-value pairs so they can be referenced in the destination template${HOSTNAME_NAME}= HOSTNAME.NAME;${HOSTNAME_ID}= HOSTNAME.ID;};destination d_file { file("/var/log/${HOSTNAME_NAME:-examplehost}/${HOSTNAME_ID}"/messages.log);};log { source(s_local); filterx(p_hostname_segmentation()); destination(d_file);};
Parse Apache log files
The following parser processes the log of Apache web servers and separates them into different fields. Apache log messages can be formatted like:
To parse such logs, the delimiter character is set to a single whitespace (delimiter=" "). Excess leading and trailing whitespace characters are stripped.
Terminal window
block filterx p_apache(){${APACHE}= json();cols=["CLIENT_IP", "IDENT_NAME", "USER_NAME",
"TIMESTAMP", "REQUEST_URL", "REQUEST_STATUS",
"CONTENT_LENGTH", "REFERER", "USER_AGENT",
"PROCESS_TIME", "SERVER_NAME"];${APACHE}= parse_csv(${MESSAGE}, columns=cols, delimiter=(" "), strip_whitespace=true, dialect="escape-double-char");# Set the important elements as name-value pairs so they can be referenced in the destination template${APACHE_USER_NAME}=${APACHE.USER_NAME};};
The results can be used for example, to separate log messages into different files based on the APACHE.USER_NAME field. in case the field is empty, the nouser string is assigned as default.
You can use multiple parsers in a layered manner to split parts of an already parsed message into further segments. The following example splits the timestamp of a parsed Apache log message into separate fields. Note that the scoping of FilterX variables is important:
If you add the new parser to the FilterX block used in the previous example, every variable is available.
If you use a separate FilterX block, only global variables and name-value pairs (variables with names starting with the $ character) are accessible from the block.
Terminal window
block filterx p_apache_timestamp(){cols=["TIMESTAMP.DAY", "TIMESTAMP.MONTH", "TIMESTAMP.YEAR", "TIMESTAMP.HOUR", "TIMESTAMP.MIN", "TIMESTAMP.SEC", "TIMESTAMP.ZONE"];${APACHE.TIMESTAMP}= parse_csv(${APACHE.TIMESTAMP}, columns=cols, delimiters=("/: "), dialect="escape-none");# Set the important elements as name-value pairs so they can be referenced in the destination template${APACHE_TIMESTAMP_DAY}=${APACHE.TIMESTAMP_DAY};};destination d_file { file("/var/log/messages-${APACHE_USER_NAME:-nouser}/${APACHE_TIMESTAMP_DAY}");};log { source(s_local); filterx(p_apache()); filterx(p_apache_timestamp()); destination(d_file);};
5.2.1 - Options of CSV parsers
The parse_csv FilterX function has the following options.
columns
Synopsis:
columns=["1st","2nd","3rd"]
Default value:
N/A
Description: Specifies the names of the columns, and correspondingly the keys in the resulting JSON array.
If the columns option is set, parse_csv returns a dictionary with the column names (as keys) and the parsed values.
If the columns option isn’t set, parse_csv returns a list.
delimiter
Synopsis:
delimiter="<string-with-delimiter-characters>"
Default value:
,
Description: The delimiter parameter contains the characters that separate the columns in the input string. If you specify multiple characters, every character will be treated as a delimiter. Note that the delimiters aren’t included in the column values. For example:
To separate the text at every hyphen (-) and colon (:) character, use delimiter="-:".
To separate the columns along the tabulator (tab character), specify delimiter="\\t".
To use strings instead of characters as delimiters, see string_delimiters.
Multiple delimiters
If you use more than one delimiter, note the following points:
AxoSyslog will split the message at the nearest possible delimiter. The order of the delimiters in the configuration file does not matter.
You can use both string delimiters and character delimiters in a parser.
The string delimiters may include characters that are also used as character delimiters.
If a string delimiter and a character delimiter both match at the same position of the input, AxoSyslog uses the string delimiter.
dialect
Synopsis:
dialect="<dialect-name>"
Default value:
escape-none
Description: Specifies how to handle escaping in the input strings.
The following values are available.
escape-backslash: The parsed message uses the backslash (\\) character to escape quote characters.
escape-backslash-with-sequences: The parsed message uses "" as an escape character but also supports C-style escape
sequences, like \n or \r. Available in AxoSyslog version 4.0 and later.
escape-double-char: The parsed message repeats the quote character when the quote character is used literally. For example, to escape a comma (,), the message contains two commas (,,).
escape-none: The parsed message does not use any escaping for using the quote character literally.
greedy
Synopsis:
greedy=true
Default value:
false
If the greedy option is enabled, AxoSyslog adds the remaining part of the message to the last column, ignoring any delimiters that may appear in this part of the message. You can use this option to process messages where the number of columns varies from message to message.
For example, you receive the following comma-separated message: example 1, example2, example3, and you segment it with the following parser:
The COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, respectively. If the message looks like example 1, example2, example3, some more information, then any text appearing after the third comma (that is, some more information) is not parsed, and thus possibly lost if you use only the parsed columns to reconstruct the message (for example, if you send the columns to different columns of a database table).
Using the greedy=true flag will assign the remainder of the message to the last column, so that the COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, some more information.
Description: List of quote pairs that are ignored and removed from the beginning and end of the strings. Note that the beginning and ending quote character does not have to be identical, for example, [} can also be a quote-pair.
In the following example, square brackets ([]) and single-quotes (') are ignored:
Terminal window
filterx {str="value1,[value2],'value3'";${MESSAGE}= parse_csv(str, quote_pairs=["[]", "'"]);# The value of ${MESSAGE} will be "value1,value2,value3"};
strip_whitespace
Synopsis:
strip_whitespace=true
Default value:
false
Description: Remove leading and trailing whitespaces from all columns. The strip_whitespace option is an alias for strip_whitespace.
Description: In case you have to use a string as a delimiter, list your string delimiters as a JSON array in the string_delimiters=["<delimiter_string1>", "<delimiter_string2>", ...] option.
By default, the parse_csv FilterX function uses the comma as a delimiter. If you want to use only strings as delimiters, you have to disable the default space delimiter, for example: delimiter="", string_delimiters=["<delimiter_string>"])
Otherwise, AxoSyslog will use the string delimiters in addition to the default character delimiter, so for example, string_delimiters=["=="] is actually equivalent to delimiters=",", string_delimiters=["=="], and not delimiters="", string_delimiters=["=="]
Multiple delimiters
If you use more than one delimiter, note the following points:
AxoSyslog will split the message at the nearest possible delimiter. The order of the delimiters in the configuration file does not matter.
You can use both string delimiters and character delimiters in a parser.
The string delimiters may include characters that are also used as character delimiters.
If a string delimiter and a character delimiter both match at the same position of the input, AxoSyslog uses the string delimiter.
5.3 - key=value pairs
The parse_kv FilterX function can split a string consisting of whitespace or comma-separated key=value pairs (for example, Postfix log messages). You can also specify other value separator characters instead of the equal sign, for example, colon (:) to parse MySQL log messages. The AxoSyslog application automatically trims any leading or trailing whitespace characters from the keys and values, and also parses values that contain unquoted whitespace.
Note
If a log message contains the same key multiple times (for example, key1=value1, key2=value2, key1=value3, key3=value4, key1=value5), then AxoSyslog only stores the last (rightmost) value for the key. Using the previous example, AxoSyslog will store the following pairs: key1=value5, key2=value2, key3=value4.
Warning
By default, the parser discards sections of the input string that are not key=value pairs, even if they appear between key=value pairs that can be parsed. To store such sections, see stray_words_key.
The names of the keys can contain only the following characters: numbers (0-9), letters (a-z,A-Z), underscore (_), dot (.), hyphen (-). Other special characters are not permitted.
If the stray_words_append_to_value flag is set, any stray words between the value pairs are appended to the preceding value. For example:
Terminal window
# input: a=b b=c d e f=gfilterx {${MESSAGE}= parse_kv(${MESSAGE}, value_separator="=", pair_separator=" ", stray_words_append_to_value=true);};# The value of $MSG will be: {"a":"b","b":"c d e","f":"g"}
If you want to collect the stray words into a separate key, see stray_words_key.
Note
Note that you cannot use stray_words_append_to_value and stray_words_key in the same parser.
stray_words_key
Specifies the key where AxoSyslog stores any stray words that appear before or between the parsed key-value pairs. If multiple stray words appear in a message, then AxoSyslog stores them as a comma-separated list. Default value:N/A
This is a list of key-value pairs, where the value separator is = and the pair separator is ;. However, before the last key-value pair (policy=370), there are two stray words: interzone-emtn_s1_vpn-enodeb_om; and inbound;. If you want to store or process these, specify a key to store them, for example:
The parse_leef FilterX function has the following options.
pair_separator
Specifies the character or string that separates the key-value pairs in the extensions. Default value: \t (tab).
LEEF v2 can specify the separator per message. Omitting this option uses the LEEF v2 provided separator, setting this value overrides it during parsing.
separate_extensions
Available in AxoSyslog 4.13 and later.
Starting with version 4.13, AxoSyslog parses fields from extensions to the same level as regular fields. In earlier versions, extensions were grouped under the extensions key. To keep using the extensions key, set separate_extensions=true.
Default value: false
value_separator
Specifies the character that separates the keys from the values in the extensions. Default value: =.
5.5 - Windows Event Log
Available in AxoSyslog 4.9 and later.
The parse_windows_eventlog_xml() FilterX function parses Windows Event Logs XMLs. It’s a specialized version of the parse_xml() parser.
The parser returns false in the following cases:
The input isn’t valid XML.
The root element doesn’t reference the Windows Event Log schema (<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>). Note that the parser doesn’t validate the input data to the schema.
For example, the following converts the input XML into a JSON object:
The parse_xml() FilterX function parses raw XMLs into dictionaries. This is a new implementation, so the limitations and options of the legacy xml-parser() do not apply.
There is no standardized way of converting XML into a dict. AxoSyslog creates the most compact dict possible. This means certain nodes will have different types and structures depending on the input XML element. Note the following points:
Empty XML elements become empty strings.
XML:<foo></foo>JSON:{"foo": ""}
Attributions are stored in @attr key-value pairs, similarly to other converters (like python xmltodict).
If an XML element has both attributes and a value, we need to store them in a dict, and the value needs a key. We store the text value under the #text key.
Add FilterX statements that select the messages you need. The following example selects messages sent by the nginx application, received from the host called example-host.
Terminal window
log {source{opentelemetry()}; filterx {# Input mappingdeclarelog= otel_logrecord(${.otel_raw.log});declareresource= otel_resource(${.otel_raw.resource});declarescope= otel_scope(${.otel_raw.scope});# FilterX statements that act as filters resource.attributes["service.name"]=="nginx"; resource.attributes["host.name"]=="example-host";}; destination {# your opentelemetry destination settings};};
To modify messages received via the OpenTelemetry protocol (OTLP), such as the ones received using the opentelemetry() source, you have to configure the following:
Map the OpenTelemetry input message to OTEL objects in FilterX, so AxoSyslog handles their type properly. Add the following to your FilterX block:
After the mapping, you can access the elements of the different data structures as FilterX dictionaries, for example, the body of the message (log.body), its attributes (log.attributes), or the attributes of the resource (resource.attributes).
The following example does two things:
It checks if the hostname resource attribute exists, and sets it to the sender IP address if it doesn’t.
It checks whether the Timestamp field (which is optional) is set in the log object, and sets it to the date AxoSyslog received the message if it isn’t.
To convert incoming syslog messages to OpenTelemetry log messages and send them to an OpenTelemetry receiver, you have to perform the following high-level steps in your configuration file:
Receive the incoming syslog messages.
Initialize the data structures required for OpenTelemetry log messages in a FilterX block.
Map the key-value pairs and macros of the syslog message to appropriate OpenTelemetry log record fields. There is no universal mapping scheme available, it depends on the source message and the receiver as well. For some examples, see the Example Mappings page in the OpenTelemetry documentation, or check the recommendations and requirements of your receiver. For details on the fields that are available in the AxoSyslog OTEL data structures, see the otel_logrecord reference.
The following example includes a simple mapping for RFC3164-formatted syslog messages. Note that the body of the message is rendered as a string, not as structured data.
Terminal window
log {source{# Configure a source to receive your syslog messages}; filterx {# Create the empty data structures for OpenTelemetry log recordsdeclarelog= otel_logrecord();declareresource= otel_resource();declarescope= otel_scope();# Set the log resource fields and map syslog values resource.attributes["host.name"]=${HOST}; resource.attributes["service.name"]=${PROGRAM}; log.observed_time_unix_nano =${R_UNIXTIME}; log.body =${MESSAGE}; log.severity_number =${LEVEL_NUM};# Update output${.otel_raw.log}= log;${.otel_raw.resource}= resource;${.otel_raw.scope}= scope;${.otel_raw.type}="log";}; destination {# your opentelemetry destination settings};};
Unique identifier of a span within a trace, an 8-byte array.
time_unix_nano
Type:
datetime
The time when the event occurred, expressed as nanoseconds elapsed since the UNIX Epoch (January 1, 1970, 00:00:00 UTC). If 0, the timestamp is missing.
trace_id
Type:
bytes
Unique identifier of a trace, a 16-byte array.
otel_resource reference
The resource describes the entity that produced the log record. It contains a set of attributes (key-value pairs) that must have unique keys. For example, it can contain the hostname and the name of the cluster.
otel_scope reference
Describes the instrumentation scope that sent the message. It may contain simple key-value pairs (strings or integers), but also arbitrary nested objects, such as lists and arrays. It usually contains a name and a version field.
Returns true if the SDATA field of the current message is not empty:
Terminal window
filterx { has_sdata();};
is_sdata_from_enterprise
Filter messages based on enterprise ID in the SDATA field. For example:
Terminal window
filterx { is_sdata_from_enterprise("6876");};
8 - Metrics
Available in AxoSyslog 4.9 and later.
You can use the update_metric function to count the processed messages, and create labeled metric counters based on the fields of the processed messages, similarly to the metrics-probe() parser.
You can configure the name of the counter to update and the labels to add. The name of the counter is an unnamed, mandatory option. Note that the name is automatically prefixed with the syslogng_ string. For example:
An integer, or an expression that resolves to an integer that defines the increment of the counter. The following example defines a counter called syslogng_input_event_bytes_total, and increases its value with the size of the incoming message (in bytes).
Note
Drivers configured with internal(yes) register their metrics on level 3. That way if you are creating an SCL, you can disable the built-in metrics of the driver, and create metrics manually using update_metric.
metrics_labels
Available in AxoSyslog 4.10 and later.
metrics_labels is a dict-like data type to store metric labels directly. You can use the metrics_labels function to convert key-values to metric labels directly. This is useful when you have multiple update_metric() function calls, because it avoids re-rendering the labels, greatly improves performance.
The stored labels are sorted alphabetically, but note that key collisions are not detected. You can use the dedup_metrics_labels() function to deduplicate labels. However, this takes CPU time, it’s better to avoid inserting keys multiple times.
9 - Update filters to FilterX
The following sections show you how you can change your existing filters and rewrite rules to FilterX statements. Note that:
Many examples in the FilterX documentation were adapted from the existing filter, parser, and rewrite examples to show how you can achieve the same functionality with FilterX.
Don’t worry if you can’t update something to FilterX. While you can’t use other blocks within a FilterX block, you can use both in a log statement, for example, you can use a FilterX block, then a parser if needed.
There is no push to use FilterX. You can keep using the traditional blocks if they satisfy your requirements.
Update filters to FilterX
This section shows you how to update your existing filter expressions to filterx.
You can replace most filter functions with a simple value comparison of the appropriate macro, for example:
facility(user) with ${FACILITY} == "user"
host("example-host") with ${HOST} == "example-host"
If you want to check for a range of levels, use numerical comparison with the ${LEVEL_NUM} macro instead. For a list of numerical level values, see LEVEL_NUM.
message("example") with ${MESSAGE} =~ "example" (see the equal tilde operator for details)
netmask() and netmask6() with subnet and a list membership check, for example, netmask(192.168.5.0/255.255.255.0) becomes ${SOURCEIP} in subnet("192.168.5.0/255.255.255.0");. For details, see IP addresses and subnets.
Since all FilterX statements must match a message to pass the FilterX block, you can often replace complex boolean filter expressions with multiple, simple FilterX statements. For example, consider the following filter statement:
Terminal window
filter { host("example1") and program("nginx");};
The following is the same FilterX statement:
Terminal window
filterx {${HOST}=="example1" and ${PROGRAM}=="nginx";};
This page describes the operators you can use in FilterX blocks.
Arithmetic operators
Available in AxoSyslog 4.12 and later.
The + (addition), - (substraction), * (multiplication), / (division), and % (modulo) operators allow you to perform arithmetic operations on numeric (integer or double) values. For example:
In version 4.18 and later, you can use the + and - operators as unary operators with a single operand to indicate a positive or a negative value. For example:
Terminal window
a= 42;b= -a;# b is -42
Plus operator
The plus operator (+) adds two arguments, if possible. (For example, you can’t add two datetime values.)
You can use it to add two numbers (two integers, two double values). If you add a double to an integer, the result is a double.
Adding two strings concatenates the strings. Note that if you want to have spaces between the added elements, you have to add them manually, like in Python, for example:
Terminal window
${MESSAGE}=${HOST} + " first part of the message," + " second part of the message" + "\n";
Adding two lists merges the lists. Available in AxoSyslog 4.9 and later.
Adding two dicts updates the dict with the values of the second operand. For example:
Terminal window
x={"key1": "value1", "key2": "value1"};y={"key3": "value1", "key2": "value2"};${MESSAGE}= x + y;# ${MESSAGE} value is {"key1": "value1", "key3": "value1", "key2": "value2"};
Available in AxoSyslog 4.9 and later.
Plus equal operator
The += operator increases the value of a variable with the value on the right. Exactly how the addition happens depends on the type of the variable.
For numeric types (int and double), the result is the sum of the values. For example:
Terminal window
a= 3;a+= 4;# a is 7b= 3.3;b+= 4.1;# b is 7.4
Adding a double value to an integer changes the integer into a double:
Terminal window
c= 3;c+= 4.1;# c is 7.1 and becomes a double
For strings (including string values in an object), it concatenates the strings. For example:
Terminal window
mystring="axo";mystring+="flow";# mystring is axoflow
For lists, it appends the new values to the list. For example:
Terminal window
mylist= json_array(["one", "two"]);mylist+=["let's", "go"];# mylist is ["one", "two", "let's", "go"]
For datetime variables, it increments the time. Note that you can add only integer and double values to a datetime, and:
When adding an integer, it must be the number of microseconds you want to add. For example:
Terminal window
d= strptime("2000-01-01T00:00:00Z", "%Y-%m-%dT%H:%M:%S%z");d+= 3600000000;# 1 hour in microseconds# d is "2000-01-01T01:00:00.000+00:00"
When adding a double, the integer part must be the number of seconds you want to add. For example:
Terminal window
d= strptime("2000-01-01T00:00:00Z", "%Y-%m-%dT%H:%M:%S%z");d+= 3600.000;# 3600 seconds, 1 hour# d is "2000-01-01T01:00:00.000+00:00"
Comparison operators
Comparison operators allow you to compare values of macros, variables, and expressions as numbers (==, <, <=, >=, >, !=) or as strings
(eq, lt, le, gt, ge, ne). You can also check for type equality (===) and strict inequality (!==). For details and examples, see Comparing values in FilterX.
Boolean operators
The not, or, and operators allow you to combine any number of comparisons and expressions. For details and examples, see Boolean operators in FilterX.
Assign if non-null (=??) operator
Available in AxoSyslog 4.10 and later.
Assigns the right operand to the left operand if the right operand exists and is not null. Note that evaluation errors of the right-hand operand will be suppressed.
Terminal window
left-operand =?? right-operand
For example:
Terminal window
resource.attributes['service.name']=?? $PROGRAM;
Using the =?? operator is equivalent to the following expression, but using =?? has better performance.
if (isset($PROGRAM) ?? false) {
resource.attributes['service.name'] = $PROGRAM;
};
Create dict element if non-null (:??) operator
Available in AxoSyslog 4.15 and later.
Creates the dict element in the left operand with the value of the right operand if the right operand exists and is not null. Note that evaluation errors of the right-hand operand will be suppressed.
For example, the following dict will have only one element, the good-field:
The null coalescing operator returns the result of the left operand if it exists and is not null, otherwise it returns the operand on the right.
Terminal window
left-operand ?? right-operand
You can use it to define a default value, or to handle errors in your FilterX statements: if evaluating the left-side operand returns an error, the right-side operand is evaluated instead.
For example, if a key of a JSON object doesn’t exist for every message, you can set it to a default value:
Terminal window
${MESSAGE}= json["BODY"] ?? "Empty message"
List membership operator
Available in AxoSyslog 4.12 and later.
You can check whether a value is in a list using the in and not in membership operators. If you’re checking the value of a variable that can have another type than the elements of a list, explicitly convert the value to the correct type (for example, string) before the comparison.
Terminal window
my_array= json_array(['hostname_one', 'hostname_two', 'hostname_three']);if(${HOST} in my_array){# ...}
Starting with AxoSyslog 4.22, you can also use the operators to check membership in dict keys. For example:
The in operator can be used to replicate the functionality of the in-list() filter function in FilterX. If you want to populate the list from a file, use the cache_json_file() FilterX function.
To check if a value contains a string or matches a regular expression, use the =~ operator. For example, the following statement is true if the ${MESSAGE} contains the word error:
Terminal window
${MESSAGE}=~ "error";
Use the !~ operator to check if a literal string or variable doesn’t contain an expression. For example, the following statement is true if the ${MESSAGE} doesn’t contain the word error:
Terminal window
${MESSAGE} !~ "error";
Note
If you want to process the matches of a search, use the regexp_search FilterX function.
If you want to rewrite or modify the matches of a search, use the regexp_subst FilterX function.
Note the following points:
Regular expressions are case sensitive by default. For case insensitive matches, add (?i) to the beginning of your pattern.
You can use regexp constants (slash-enclosed regexps) within FilterX blocks to simplify escaping special characters, for example, /^beginning and end$/.
FilterX regular expressions are interpreted in “leave the backslash alone mode”, meaning that a backslash in a string before something that doesn’t need to be escaped and will be interpreted as a literal backslash character. For example, string\more-string is equivalent to string\\more-string.
String slicing (..)
Available in AxoSyslog 4.15 and later.
You can slice strings at the specified index using the .. operator to get a section of the string. Indexing starts at 0. You can omit the index to refer to the beginning or the end of the string. For example:
Terminal window
filterx {str="example";idx= 3;my_string= str[idx..5];# Value of my_string is "mp";my_string= str[..idx];# Value of my_string is "exa";my_string= str[idx..];# Value of my_string is "mple";};
Staring with AxoSyslog version 4.17, you can use negative indexes to refer to characters from the end of the string, for example:
The ternary conditional operator evaluates an expression and returns the first argument if the expression is true, and the second argument if it’s false.
For example, the following example checks the value of the ${LEVEL_NUM} macro and returns low if it’s lower than 5, high otherwise.
Terminal window
(${LEVEL_NUM} < 5) ? "low" : "high";
You can also use it to check if a value is set, and set it to a default value if it isn’t, but for this use case we recommend using the Null coalescing operator:
This page describes the functions you can use in FilterX blocks.
Functions have arguments that can be either mandatory or optional.
Mandatory options are always positional, so you need to pass them in the correct order. You cannot set them in the arg=value format.
Optional arguments are always named, like arg=value. You can pass optional arguments in any order.
base64_decode
Available in AxoSyslog 4.25 and later.
Decodes a Base64-encoded string and returns the result as a bytes value.
Usage: base64_decode(string)
For example:
Terminal window
base64_decode("Zm9vYmFy");# Returns the bytes "foobar"base64_decode(base64_encode("szilvafa"));# Round-trips back to the original value
base64_encode
Available in AxoSyslog 4.25 and later.
Encodes a string or bytes value as a Base64 string.
Usage: base64_encode(string_or_bytes)
For example:
Terminal window
base64_encode("foobar");# Returns "Zm9vYmFy"
cache_json_file
Load the contents of an external JSON file in an efficient manner. You can use this function to lookup contextual information. (Basically, this is a FilterX-specific implementation of the add-contextual-data() functionality.)
AxoSyslog automatically detects if the file is updated and reloads the file.
Then the following FilterX expression selects only “web” traffic:
Terminal window
filterx {declareknown_apps= cache_json_file("/context-info-db.json");${app}= known_apps[${PROGRAM}] ?? "unknown";${app}=="web";# drop everything that's not a web server log}
To avoid failures when the file might not yet exist, provide a default_value:
Usage: datetime(<string or expression to cast as datetime>)
For example:
Terminal window
date= datetime("1701350398.123000+01:00");
Usually, you use the strptime FilterX function to create datetime values. Alternatively, you can cast an integer, double, string, or isodate variable into datetime with the datetime() FilterX function. Note that:
When casting from an integer, the integer is the number of microseconds elapsed since the UNIX epoch (00:00:00 UTC on 1 January 1970).
When casting from a double, the double is the number of seconds elapsed since the UNIX epoch (00:00:00 UTC on 1 January 1970). (The part before the floating points is the seconds, the part after the floating point is the microseconds.)
When casting from a string, the string (for example, 1701350398.123000+01:00) is interpreted as: <the number of seconds elapsed since the UNIX epoch>.<microseconds>+<timezone relative to UTC (GMT +00:00)>
dedup_metrics_labels
Deduplicate metrics_labels objects. For details, see metrics_labels.
my_dict={"key_1": "value_1",
"key_2": "value_2",
"key_3": ["value_3", "value_4"],
};my_list= dict_to_pairs(my_dict, "key", "value");# The value of my_list will be:# [# {"key":"key_1","value":"value_1"},# {"key":"key_2","value":"value_2"},# {"key":"key_3","value":["value_3","value_4"]}# ]
digest
Available in AxoSyslog 4.25 and later.
Computes a cryptographic hash of a string or bytes value and returns the raw hash as a bytes object. Use the optional alg= argument to select the hash algorithm. If alg= is not set, the default is sha256. The algorithm name is passed to OpenSSL, so any name accepted by EVP_get_digestbyname is supported (for example, md5, sha1, sha256, sha512). An unknown algorithm name causes a configuration error.
To obtain the hash as a hexadecimal string instead of bytes, use the convenience functions md5, sha1, sha256, or sha512.
Usage: digest(string_or_bytes, alg="sha256")
For example:
Terminal window
digest("foobar");# Raw SHA-256 hash as bytesdigest("foobar", alg="md5");# Raw MD5 hash as bytes
dpath
Available in AxoSyslog 4.17 and later.
Assigns a value to a dictionary and creates any elements of the path that don’t exist. For example:
Flattens the nested elements of an object using the specified separator, similarly to the format-flat-json() template function. For example, you can use it to flatten nested JSON objects in the output if the receiving application cannot handle nested JSON objects.
Usage: flatten(dict_or_list, separator=".")
You can use multi-character separators, for example, =>. If you omit the separator, the default dot (.) separator is used.
Formats a date as ISODATE: %Y-%m-%dT%H:%M:%S%z. For example:
Terminal window
my_date= strptime("2000-01-02T03:04:05.678901-07:00", "%Y-%m-%dT%H:%M:%S.%f%Z");${MESSAGE}= format_isodate(my_date);# The value of ${MESSAGE} is 2000-01-02T03:04:05.678901-07:00
Formats a dictionary into Windows Event Logs XML. It’s a specialized version of the format_xml() function, all generic formatting tips apply to format_windows_eventlog_xml() as well.
Matches a filename (or any string) against one or more glob patterns and returns true if the filename matches any of the patterns, false otherwise. Note that / separators are matched literally (not by wildcards), and a leading . must be matched explicitly.
Usage: glob_match(filename, patterns)
filename: The string to test.
patterns: A single pattern string, or a list of pattern strings. When a list is provided, the function returns true as soon as any pattern matches.
Using a non-string filename, or a non-string element in the patterns list, causes a runtime error.
Decodes a lowercase or uppercase hexadecimal string and returns the result as bytes. The input length must be even, and each character must be a valid hexadecimal digit. Invalid characters or an odd-length input cause a runtime error.
Usage: hex_decode(string)
For example:
Terminal window
hex_decode("666f6f626172");# Returns the bytes "foobar"
hex_encode
Available in AxoSyslog 4.25 and later.
Encodes a string or bytes value as a lowercase hexadecimal string.
Usage: hex_encode(string_or_bytes)
For example:
Terminal window
hex_encode("foobar");# Returns "666f6f626172"
includes
Available in AxoSyslog 4.9 and later.
Returns true if the input string contains the specified substring. By default, matches are case sensitive. Usage:
If the object doesn’t exist, istype() returns with an error, causing the FilterX statement to become false, and logs an error message to the internal() source of AxoSyslog.
json
Cast a value into a JSON object.
Usage: json(<string or expression to cast to json>)
For example:
Terminal window
js_dict= json({"key": "value"});
Starting with version 4.9, you can use {} without the json() keyword as well. For example, the following creates an empty JSON object:
Terminal window
js_dict={};
json_array
Cast a value into a JSON array.
Usage: json_array(<string or expression to cast to json array>)
Starting with version 4.9, you can use [] without the json_array() keyword as well. For example, the following creates an empty JSON list:
Terminal window
js_dict=[];
keys
Returns the top-level keys of a dictionary. This provides a simple way to inspect or iterate over the immediate keys without traversing the structure. The keys() function:
Returns a list of dictionary keys as an array.
Includes only the top-level keys, ignoring nested structures.
The resulting array supports immediate indexing for quick key retrieval.
When called on an empty dictionary, keys returns an empty dictionary ([]).
For example:
Terminal window
dict={"level1-key1":{"level2-key1":{"level3-key1":"value1"}},"level1-key2":{"level2-key2":{"level3-key2":"value2"}}};# accessing the top level, returns: ["level1-key1", "level1-key2"]a= keys(dict);# accessing nested levels directly, returns: ["level2-key1"]b= keys(dict["level1-key1"]);# directly index the result of keys() to access specific keys is possible, returns: ["level1-key1"])c= keys(dict)[0];
len
Returns the number of items in an object as an integer: the length (number of characters) of a string, the number of elements in a list, or the number of keys in an object.
Usage: len(object)
load_vars
Loads variables from a dict. It’s the inverse of vars(). It loads and declares FilterX-level variables. If a key in the dict begins with the $ character, it’s loaded as an AxoSyslog macro. This function can be used to transfer several variables between FilterX blocks on different log paths and messages.
lower
Converts all characters of a string lowercase characters.
Usage: lower(string)
md5
Available in AxoSyslog 4.25 and later.
Computes the MD5 hash of a string or bytes value and returns the result as a lowercase hexadecimal string. To obtain the raw hash as bytes, use the digest function with alg="md5".
Convert key-values to metric labels directly. For details, see metrics_labels.
move
Available in AxoSyslog 4.23 and later.
Moves the specified variable to its new location, instead of copying it. This is equivalent to using a value assignment and an unset function, but has better performance. The following example moves the JSON object old into the new.nest field:
Searches a string and returns the matches of a regular expression as a list or a dictionary. If there are no matches, the result is empty.
Note
In version 4.9 and earlier, regexp_search returned a dict or list depending on whether named match groups were used in the expression. Starting with version 4.10, dict is returned by default. For details, see list_mode.
Match group zero is now excluded by default unless it’s the only match group. To always include the zero match group in the results, use the keep_zero=true flag.
You can also use unnamed match groups (()) and named match groups ((?<first>ERROR)(?<second>message)).
Note the following points:
Regular expressions are case sensitive by default. For case insensitive matches, add (?i) to the beginning of your pattern.
You can use regexp constants (slash-enclosed regexps) within FilterX blocks to simplify escaping special characters, for example, /^beginning and end$/.
FilterX regular expressions are interpreted in “leave the backslash alone mode”, meaning that a backslash in a string before something that doesn’t need to be escaped and will be interpreted as a literal backslash character. For example, string\more-string is equivalent to string\\more-string.
Options
You can use the following optional flags in regexp_search:
keep_zero: Always return the zero match group. Available in version 4.10 and later. Default value: false
list_mode: Return results as a list. Available in version 4.10 and later. Default value: false
If the result is an existing dict or list object, the function respects the type of the object, even if list_mode is set to true.
${MY-LIST}.named is a dictionary with the names of the match groups as keys, and the corresponding matches as values: {"0": "first-word second-part third", "one": "first-word", "two": "second-part", "three": "third"},
Mixed match groups
If you use mixed (some named, some unnamed) groups in your regular expression, the output is a dictionary, where AxoSyslog automatically assigns a key to the unnamed groups. For example:
Rewrites a string using regular expressions. This function implements the subst rewrite rule functionality. If you need only string replacement without regular expression support, use the str_replace function as it has better performance.
Regular expressions are case sensitive by default. For case insensitive matches, add (?i) to the beginning of your pattern.
You can use regexp constants (slash-enclosed regexps) within FilterX blocks to simplify escaping special characters, for example, /^beginning and end$/.
FilterX regular expressions are interpreted in “leave the backslash alone mode”, meaning that a backslash in a string before something that doesn’t need to be escaped and will be interpreted as a literal backslash character. For example, string\more-string is equivalent to string\\more-string.
Starting with version 4.10 substitution match groups is enabled by default (use the groups=false flag to disable that if needed). You can reference match group indexes up to 999.
When configured, it changes the newline definition used in PCRE regular expressions to accept either of the following:
a single carriage-return
linefeed
the sequence carriage-return and linefeed (\\r, \\n and \\r\\n, respectively)
This newline definition is used when the circumflex and dollar patterns (^ and $) are matched against an input. By default, PCRE interprets the linefeed character as indicating the end of a line. It does not affect the \\r, \\n or \\R characters used in patterns.
utf8=true:
Use Unicode support for UTF-8 matches: UTF-8 character sequences are handled as single characters.
set_fields
Takes a dict and sets multiple fields in it with overrides or defaults (overrides and defaults are optional parameters).
The overrides and defaults parameters are also dicts, where:
the key is the field’s name
the value is either an expression, or a list of expressions.
If a list is provided, each expression will be evaluated, and the first successful, non-null one is set as the respective field’s value. This is similar to chaining null-coalescing (??) operators, but has better performance.
overrides are always processed for each field. The defaults option for a field is only processed if the field isn’t set, or it’s empty.
Computes the SHA-1 hash of a string or bytes value and returns the result as a lowercase hexadecimal string. To obtain the raw hash as bytes, use the digest function with alg="sha1".
Computes the SHA-256 hash of a string or bytes value and returns the result as a lowercase hexadecimal string. To obtain the raw hash as bytes, use the digest function (sha256 is also the default algorithm used by digest when alg= is not set).
Computes the SHA-512 hash of a string or bytes value and returns the result as a lowercase hexadecimal string. To obtain the raw hash as bytes, use the digest function with alg="sha512".
Usage: sha512(string_or_bytes)
For example:
Terminal window
sha512("foobar");
startswith
Available in AxoSyslog 4.9 and later.
Returns true if the input string begins with the specified substring. By default, matches are case sensitive. Usage:
You can use the following format codes in the format string:
%a: The locale’s abbreviated weekday name.
%A: The locale’s full weekday name.
%b: The locale’s abbreviated month name.
%B: The locale’s full month name.
%c: The locale’s appropriate date and time representation.
%C: The year divided by 100 and truncated to an integer, as a decimal number.
%d: The day of the month as a decimal number [01,31].
%D: Equivalent to %m / %d / %y.
%e: The day of the month as a decimal number [1,31]; a single digit is preceded by a space.
%f: Fraction of the second (with or without a leading dot). Width specifies precision, %6f means microseconds, %3f means milliseconds, %9f means nanoseconds. %f just means microseconds.
%F: Equivalent to %+4Y-%m-%d.
%g: The last 2 digits of the week-based year (see below) as a decimal number [00,99].
%G: The week-based year (see below) as a decimal number (for example, 1977).
%h: Equivalent to %b.
%H: The hour (24-hour clock) as a decimal number [00,23].
%I: The hour (12-hour clock) as a decimal number [01,12].
%j: The day of the year as a decimal number [001,366].
%m: The month as a decimal number [01,12].
%M: The minute as a decimal number [00,59].
%n: A <newline>.
%p: The locale’s equivalent of either a.m. or p.m.
%r: The time in a.m. and p.m. notation.
%R: The time in 24-hour notation ( %H : %M ).
%S: The second as a decimal number [00,60].
%t: A <tab>.
%T: The time (%H : %M : %S).
%u: The weekday as a decimal number [1,7], with 1 representing Monday.
%U: The week number of the year as a decimal number [00,53]. The first Sunday of January is the first day of week 1; days in the new year before this are in week 0.
%V: The week number of the year (Monday as the first day of the week) as a decimal number [01,53]. If the week containing 1 January has four or more days in the new year, then it is considered week 1. Otherwise, it is the last week of the previous year, and the next week is week 1. Both January 4th and the first Thursday of January are always in week 1.
%w: The weekday as a decimal number [0,6], with 0 representing Sunday.
%W: The week number of the year as a decimal number [00,53]. The first Monday of January is the first day of week 1; days in the new year before this are in week 0.
%x: The locale’s appropriate date representation.
%X: The locale’s appropriate time representation.
%y: The last two digits of the year as a decimal number [00,99].
%Y: The year as a decimal number (for example, 1997).
%z: The offset from UTC in the ISO 8601:2000 standard format ( +hhmm or -hhmm ), or by no characters if no timezone is determinable
%Z: Same as %z , but with the : separator (-hh:mm or +hh:mm)
string
Cast a value into a string. Note that currently AxoSyslog evaluates strings and executes template functions and template expressions within the strings. In the future, template evaluation will be moved to a separate FilterX function.
Usage: string(<string or expression to cast>)
For example:
Terminal window
myvariable= string(${LEVEL_NUM});
Sometimes you have to explicitly cast values to strings, for example, when you want to concatenate them into a message using the + operator.
str_replace
Available in AxoSyslog 4.15 and later.
Replace a literal string with another one. If you need to use regular expressions, see regexp_subst.
If you don’t specify the max occurrence, every match is replaced, otherwise only the first <max-occurrence>. For example:
Terminal window
filterx {my_input="This is an input string";my_input= str_replace(my_input, "in", "out");# Value of my_input becomes: "This is an output stroutg"my_input="This is an input string";my_input= str_replace(my_input, "in", "out", 1);# Value of my_input becomes: "This is an output string"};
Note that the search is case sensitive, and supports UTF-8 characters.
str_strip, str_lstrip, str_rstrip
Available in AxoSyslog 4.16 and later.
These functions remove the leading and/or trailing whitespaces from a string, including the \n\r\t characters.
strptime
Creates a datetime object from a string, similarly to the date-parser(). The first argument is the string containing the date. The second argument is a format string that specifies how to parse the date string. Optionally, you can specify additional format strings that are applied in order if the previous one doesn’t match the date string.
Note
If none of the format strings match, strptime returns the null value and logs an error message to the internal() source of AxoSyslog. If you want the FilterX block to explicitly return false in such cases, use the isset FilterX function on the result of strptime.
You can use the following format codes in the format string:
%a: The day of week, using the locale’s weekday names. Either the abbreviated or full name may be specified.
%A: Same as %a.
%b: The month, using the locale’s month names. Either the abbreviated or full name may be specified.
%B: The same as %b.
%c: The date and time, using the locale’s date and time format.
%C: The century number [0,99]. Leading zeros are permitted but not required. This conversion should be used in conjunction with the %y conversion.
%d: The day of month [1,31]. Leading zeros are permitted but not required.
%D: The date as %m/%d/%y.
%e: The same as %d.
%F: The date as %Y-%m-%d (the ISO 8601 date format).
%g: The year corresponding to the ISO week number, without the century. (A NetBSD extension.)
%G: The year corresponding to the ISO week number, with the century. (A NetBSD extension.)
%h: The same as %b.
%H: The hour (24-hour clock) [0,23]. Leading zeros are permitted but not required.
%I: The hour (12-hour clock) [1,12]. Leading zeros are permitted but not required.
%j: The day number of the year [1,366]. Leading zeros are permitted but not required.
%k: The same as %H.
%l: The same as %I.
%m: The month number [1,12]. Leading zeros are permitted but not required.
%M: The minute [0,59]. Leading zeros are permitted but not required.
%n: Any white-space, including none.
%p: The locale’s equivalent of a.m. or p.m.
%r: The time (12-hour clock) with %p, using the locale’s time format.
%R: The time as %H:%M.
%S: The seconds [0,60]. Leading zeros are permitted but not required.
%s: The number of seconds since the Epoch, UTC (see mktime(3)). (A NetBSD extension.)
%f: Fraction of the second (with or without a leading dot).
%t: Any white-space, including none.
%T: The time as %H:%M:%S.
%u: The day of the week as a decimal number, where Monday = 1. (A NetBSD extension.)
%U: The week number of the year (Sunday as the first day of the week) as a decimal number [0,53]. Leading zeros are permitted but not required. All days in a year preceding the first Sunday are considered to be in week 0.
%V: The ISO 8601:1988 week number as a decimal number. If the week (starting on Monday) that contains January 1 has more than three days in the new year, then it is considered the first week of the year. If it has fewer than four days in the new year, then it is considered the last week of the previous year. Weeks are numbered from 1 to 53. (A NetBSD extension.)
%w: The weekday as a decimal number [0,6], with 0 representing Sunday. Leading zeros are permitted but not required.
%W: The week number of the year (Monday as the first day of the week) as a decimal number [0,53]. Leading zeros are permitted but not required. All days in a year preceding the first Monday are considered to be in week 0.
%x: The date, using the locale’s date format.
%X: The time, using the locale’s time format.
%y: The year within the 20th century [69,99] or the 21st century [0,68]. Leading zeros are permitted but not required. If specified in conjunction with %C, specifies the year [0,99] within that century.
%Y: The year, including the century (i.e., 1996).
%Z: Timezone in ascii format (for example, PST), or in format -/+0000, accepts : in the middle of timezones (ISO 8601)
%z: Timezone in ascii format (for example, PST), or in format -/+0000, accepts : in the middle of timezones (ISO 8601) (Required element)
%%: matches a literal %. No argument is converted.
Warning
When using the %z and %Z format codes, consider that while %z strictly expects a specified timezone, and triggers a warning if the timezone is missing, %Z does not trigger a warning if the timezone is not specified.
Deletes (unsets) the empty fields of an object, for example, a JSON object or list. By default, the object is processed recursively, so the empty values are deleted from inner dicts and lists as well. If you set the replacement option, you can also use this function to replace fields of the object to custom values.
Usage: unset_empties(object, options)
The unset_empties() function has the following options:
ignorecase: Set to true to perform case-insensitive matching. Default value: false. Available in AxoSyslog 4.9 and later, default changed to false in version 4.10.
recursive: Enables recursive processing of nested dictionaries. Default value: true
replacement: Replace the target elements with the value of replacement instead of removing them. Available in AxoSyslog 4.9 and later.
targets: A list of elements to remove or replace. Default value: ["", null, [], {}]. Available in AxoSyslog 4.9 and later.
For example, to remove the fields with - and N/A values, you can use
Returns a string where invalid UTF-8 byte sequences in the input are replaced with their \xNN escaped representation. If the input is already valid UTF-8, the original string is returned unchanged. The function is idempotent: calling it on an already sanitized value produces the same result.
Generates a random RFC 9562 UUIDv7 identifier, which embeds a millisecond-precision Unix timestamp, so they sort lexically by creation time.
Usage:
Terminal window
uuid7()
For example:
Terminal window
${MESSAGE}= string(uuid7());
vars
Returns the variables (including pipeline variables and name-value pairs) defined in the FilterX block as a JSON object. The names of name-value pairs begins with the $ character. To exclude name-value pairs, set the exclude_msg_values=true flag.
Corrects the timezone of a message if it was parsed incorrectly for some reason, or if the client didn’t include any timezone information in the message. For example:
Attempts to set the timezone of the message automatically, using heuristics on the timestamps. Normally AxoSyslog performs this operation automatically when it parses the incoming message. Use this function if you can’t parse the incoming message for some reason, but you want to set the timezone automatically, for example, after you have preprocessed the message. Using this function is identical to using the flags(guess-timezone) flag in the source.
Sets the timezone of the message to a specific value, or converts an existing timezone to a different one. This operation is identical to setting the time-zone() option in the destination or as a global option, but can be applied selectively to the messages using conditions.
FilterX has two types to work with network addresses:
ip() represents a single IPv4 or IPv6 address, while
subnet() represents an IPv4 or IPv6 subnet in CIDR notation.
You can use these types together with the list membership operator (in) to check whether an address belongs to a subnet, which is useful for filtering messages based on the source or destination network of the traffic they describe.
ip
The ip() type represents a single IPv4 or IPv6 address as a string. AxoSyslog returns an error if the argument cannot be parsed as a valid IP address.
You can also pass a variable that contains the address as a string:
Terminal window
filterx {src="192.168.2.10";addr= ip(src);};
subnet
The subnet() type represents an IPv4 or IPv6 subnet as a string in CIDR notation. AxoSyslog returns an error if the argument cannot be parsed as a valid subnet.
For IPv4, you can use either prefix length notation (for example, /24) or a full netmask (for example, /255.255.255.0).
For IPv6, only prefix length notation is accepted (for example, /64).
If the address part contains host bits, AxoSyslog masks them out so the resulting subnet always represents the network address. For example, subnet("192.0.2.5/24") is equivalent to subnet("192.0.2.0/24").
To check whether an IP address belongs to a subnet, use the list membership operator (in). The left-hand side can be a string that contains an IP address, or an ip() object. The right-hand side must be a subnet() object.
The following example checks whether a string IP address is in an IPv4 subnet:
Terminal window
filterx {net= subnet("192.0.2.0/24");result= json(); result.member ="192.0.2.100" in net;# true result.non_member ="198.51.100.1" in net;# false${MESSAGE}= result;};
The same works for IPv6 subnets:
Terminal window
filterx {net= subnet("2001:db8::/32");result= json(); result.member ="2001:db8::1" in net;# true result.non_member ="2001:db9::1" in net;# false${MESSAGE}= result;};
You can also use an ip() object on the left-hand side, which is useful when the address is already stored as an ip() value:
Terminal window
filterx {net= subnet("192.0.2.0/24");addr= ip("192.0.2.100"); addr in net;# true};
CAUTION:
The address family of the IP and the subnet must match. Comparing an IPv4 address to an IPv6 subnet (or the other way around) evaluates to false, so the surrounding statement is treated as falsy and the message is dropped from the log path. Wrap the check in a conditional if you want to keep processing the message in this case.
Filter messages based on the source network
A typical use case is to filter or tag messages based on the network the message originated from. The following example tags messages that come from the 192.168.2.0/24 internal network:
Terminal window
filterx {internal_net= subnet("192.168.2.0/24");${labels}= json();if(${SOURCEIP} in internal_net){${labels.network}="internal";}else{${labels.network}="external";};};
You can combine multiple subnets with logical operators to match a list of allowed networks:
Terminal window
filterx {office= subnet("192.0.2.0/24");datacenter= subnet("198.51.100.0/24");${labels}= json();if(${SOURCEIP} in office or ${SOURCEIP} in datacenter){${labels.network}="trusted";};};
15 - Troubleshooting
To help troubleshooting FilterX blocks, AxoSyslog includes some specific functions that allow you to track failures in FilterX code:
failure_info_enable(): Collect failure information from this point downwards through all branches of the pipeline. By default, only truthy expressions are collected. To collect collect falsy evaluations as well, use failure_info_enable(collect_falsy=true);
failure_info_clear(): Clear all failure information collected so far.
failure_info_meta({}): Attach metadata to the given section of FilterX code. The metadata remains in effect until the next call, or until the end of the enclosing FilterX block, whichever comes first. For example, you can use this function to mark where you are in a decision tree:
failure_info(): Return the collected failure information as a FilterX dictionary. Call this function as late as possibly, for example, in the last log path of your AxoSyslog configuration, or within a fallback path. The output looks like:
[{"meta":{"step":"Setting common fields"},"location":"/etc/syslog-ng/syslog-ng.conf:33:7","line":"nonexisting.key = 13;","error":"No such variable: nonexisting"}]
The following is an example configuration that uses these functions:
Terminal window
destination console_output { stdout(template("$a\n"));};source input { channel {source{ stdin(); network(port(4444));}; filterx {# it can be enabled for a subset of messages failure_info_enable(collect_falsy=true);};};};log { source(input); log "log-path-1"{ filterx {# "log-path-1" was successful, clear accumulated errors failure_info_clear();};}; log "log-path-2"{ filterx {# Step #1: abc failure_info_meta({"step": "#1 log-path-2"});a= 1;b= 3;1== 1; true;# Step #2: cba failure_info_meta({"step": "#2 log-path-2"});declareg= 33; nonexisting.key = g;};}; log "log-path-3"{ filterx { failure_info_meta({"step": "falsystep"});1== 0; true;};};};# WARNING: use the last logpath in the config file, or a real fallback pathlog "fallback"{ source(input); filterx {$a= failure_info();}; destination(console_output);};
If you start AxoSyslog with this configuration, the output will look like this (because it’s trying to assign a value to a non-existing variable):
[{"meta":{"step":"Setting common fields"},"location":"/etc/syslog-ng/syslog-ng.conf:33:7","line":"nonexisting.key = 13;","error":"No such variable: nonexisting"}]
16 - Format data
AxoSyslog FilterX has several functions to format data into specific formats.
16.1 - Comma-separated values
Formats a dictionary or a list into a comma-separated string.
Only the input is mandatory, other arguments are optional. Note that the delimiter must be a single character.
By default, the delimiter is the comma (delimiter=","), the columns and default_value are empty.
If the columns option is set, AxoSyslog checks that the number of fields or entries in the input data matches the number of columns. If there are fewer items, it adds the default_value to the missing entries.
The following keys must be available in the dictionary, otherwise formatting fails with an error message like: FILTERX ERROR; ....| format_cef(my_dictionary)', error='Failed to evaluate event formatter function:.
The value_separator option must be a single character, the pair_separator can be a string. For example, to use the colon (:) as the value separator and the semicolon (;) as the pair separator, use:
The following keys must be available in the dictionary, otherwise formatting fails with an error message like: FILTERX ERROR; ....| format_leef(my_dictionary)', error='Failed to evaluate event formatter function:.
LEEF version 1: leef_version, vendor_name, product_name, product_version, event_id, extensions
LEEF version 2: leef_version, vendor_name, product_name, product_version, event_id, leef_delimiter, extensions
Setting the message option is required. You can set the other options using any FilterX variable, function, or expression. If you specify a nonexisting variable, or if evaluating an expression fails, default values will be used.
If you want to include SDATA in the message, set the SDATA macro.
<13>1 2025-12-07T22:34:32.000000+00:00 host-value prog-value 54241234 - My static message text
16.7 - Windows Event Logs XML
Available in AxoSyslog 4.13 and later.
Formats a dictionary into Windows Event Logs XML. It’s a specialized version of the format_xml() function, all generic formatting tips apply to format_windows_eventlog_xml() as well.