Processing steps

Flows can include a list of processing steps to select and transform the messages. All log messages received by the routers of the flow are fed into the pipeline constructed from the processing steps, then the filtered and processed messages are forwarded to the specified destination.

To add processing steps to an existing flow, complete the following steps.

1. Open the Flows page and select the flow you want to modify.

2. Select Process > Add new Processing Step.

3. Select the type of the processing step to add.

   

   The following types of processing steps are available:

   • Select Messages: Filter the messages using a query. Only the matching messages will be processed in subsequent steps.
   • Set Field: Set the value of a field on the message object.
   • Unset Field: Unset the specified field of the message object.
   • Reduce: Automatically remove redundant content from the messages.
   • Regex: Execute a regular expression on the message.
   • FilterX: Execute an arbitrary Filterx snippet.
   • Probe: A measurement point that provides metrics about the throughput of the flow.

4. Configure the processing step as needed for your environment.

5. (Optional) To apply the processing step only to specific messages from the flow, set the Condition field of the step. Conditions act as filters, but apply only to this step. Messages that don’t match the condition will be processed by the subsequent steps.

6. (Optional) If needed, drag-and-drop the step to change its location in the flow.

7. Select Save.

FilterX

Execute an arbitrary FilterX snippet. The following example checks if the meta.host.labels.team field exists, and sets the meta.openobserve variable to a JSON value that contains stream as a key with the value of the meta.host.labels.team field.

Probe

Set a measurement point that provides metrics about the throughput of the flow. Note that you can’t name your probes input and output, as these are reserved.

For details, see Flow metrics.

Reduce

Automatically remove redundant content from the messages.

Regex

Use a regular expression to process the message.

• Input is an AxoSyslog template string, for example, ${MESSAGE} that the regular expression will be matched with.

• Pattern is a Perl Compatible Regular Expression (PCRE). If you want to match the pattern to the whole string, use the ^ and $ anchors at the beginning and end of the pattern.

  You can use https://regex101.com/ with the ECMAScript flavor to validate your regex patterns. (When denoting named capture groups, ?P is not supported, use ?.)

Named or anonymous capture groups denote the parts of the pattern that will be extracted to the Field field. Named capture groups can be specified like (?<group_name>pattern), which will be available as field_name["group_name"]. Anonymous capture groups will be numbered, for example, the second group from (first.*) (bar|BAR) will be available as field_name["2"]. When specifying variables, note the following points:

• If you want to include the results in the outgoing message or in its metadata for other processing steps, you must:

  • Reference an existing variable or field. This field will be updated with the matches.
  • For new variables, use names for the variables that start with the $ character, or
  • declare them explicitly. For details, see the FilterX data model in the AxoSyslog documentation.

• The field will be set to an empty dictionary ({}) when the pattern doesn’t match.

For example, the following processing step parses an unclassified log message from a firewall into a structured format in the log.body field:

• Input: ${RAWMSG}

• Pattern:

  ^(?<priority><\d+>)(?<version>\d+)\s(?<timestamp>[\d\-T:.Z]+)\s(?<hostname>[\w\-.]+)\s(?<appname>[\w\-]+)\s(?<event_id>\d+|\-)\s(?<msgid>\-)\s(?<unused>\-)\s(?<rule_id>[\w\d]+)\s(?<network>\w+)\s(?<action>\w+)\s(?<result>\w+)\s(?<rule_seq>\d+)\s(?<direction>\w+)\s(?<length>\d+)\s(?<protocol>\w+)\s(?<src_ip>\d+\.\d+\.\d+\.\d+)\/(?<src_port>\d+)->(?<dest_ip>\d+\.\d+\.\d+\.\d+)\/(?<dest_port>\d+)\s(?<tcp_flags>\w+)\s(?<policy>[\w\-.]+)
  

• Field: log.body

Select messages

Filter the messages using a query. Only the matching messages will be processed in subsequent steps.

The following example selects messages that have the resource.attributes["service.name"] label set to nginx.

You can also select messages using various other metadata about the connection and the host that sent the data, the connector that received the data, the classification results, and also custom labels, for example:

• a specific connector: meta.connector.name = axorouter-soup
• a type of connector: meta.connector.type = otlp
• a sender IP address: meta.connection.src_ip = 192.168.1.1
• a specific product: meta.product = fortigate (see Vendors for the metadata of a particular product)
• a custom label you’ve added to the host: meta.host.labels.location = eu-west-1

In addition to exact matches, you can use the following operators:

• != not equal (string match)
• =* contains (substring match)
• !*: doesn’t contain
• =~: matches regular expression
• !~: doesn’t match regular expression
• ==~: matches case sensitive regular expression
• !=~: doesn’t match case sensitive regular expression

You can combine multiple expressions using the AND (+) operator. (To delete an expression, hit SHIFT+Backspace.)

Set field

Set specific field of the message. You can use static values, and also dynamic values that were extracted from the message automatically by Axoflow or manually in a previous processing step. If the field doesn’t exist, it’s automatically created. The following example sets the resource.attributes.format field to parsed.

Unset field

Unset a specific field of the message.