Processing steps

Flows can include a list of processing steps to select and transform the messages. All log messages received by the routers of the flow are fed into the pipeline constructed from the processing steps, then the filtered and processed messages are forwarded to the specified destination.

To add processing steps to an existing flow, complete the following steps.

1. Open the Flows page and select the flow you want to modify.

2. Select Process > Add new Processing Step.

3. Select the type of the processing step to add.

   

   The following types of processing steps are available:

   • Select Messages: Filter the messages using a query. Only the matching messages will be processed in subsequent steps.
   • Set Field: Set the value of a field on the message object.
   • Unset Field: Unset the specified field of the message object.
   • Reduce: Automatically remove redundant content from the messages.
   • FilterX: Execute an arbitrary Filterx snippet.
   • Probe: A measurement point that provides metrics about the throughput of the flow.

4. Configure the processing step as needed for your environment.

5. (Optional) To apply the processing step only to specific messages from the flow, set the Condition field of the step. Conditions act as filters, but apply only to this step. Messages that don’t match the condition will be processed by the subsequent steps.

6. (Optional) If needed, drag-and-drop the step to change its location in the flow.

7. Select Save.

FilterX

Execute an arbitrary Filterx snippet. The following example checks if the meta.host.labels.team field exists, and sets the meta.openobserve variable to a JSON value that contains stream as a key with the value of the meta.host.labels.team field.

Probe

Set a measurement point that provides metrics about the throughput of the flow.

For details, see Flow metrics.

Reduce

Automatically remove redundant content from the messages.

Select messages

Filter the messages using a query. Only the matching messages will be processed in subsequent steps. The following example selects messages that have the resource.attributes["service.name"] label set to nginx.

Set field

Set specific field of the message. You can use static values, and also dynamic values that were extracted from the message automatically by Axoflow or manually in a previous processing step. If the field doesn’t exist, it’s automatically created. The following example sets the resource.attributes.format field to parsed.

Unset field

Unset a specific field of the message.