Data management

• 1: Processing elements
• 2: Flow management
• 2.1: Flow overview
• 2.2: Create flow
• 2.3: Processing steps
• 2.4: Flow metrics
• 2.5: Flow analytics
• 2.6: Paths
• 2.7: Flow tapping

1 - Processing elements

Axoflow processes the data transported in your security data pipeline in the following stages:

1. Sources: Data enters the pipeline from a data source. A data source can be an external appliance or application, or a log collector agent managed by Axoflow.

2. Custom metadata on the source: You can configure Axoflow to automatically add custom metadata to the data received from a source.

3. Router: The AxoRouter data aggregator processes the data it receives from the sources:

   • Connector: AxoRouter hosts receive data using source connectors. The different connectors are responsible for different protocols (like Syslog or OpenTelemetry). Some metadata labels are added to the data based on the connector it was received.
   • Metadata: AxoRouter classifies and identifies the incoming messages and adds metadata, for example, the vendor and product of the identified source.
   • Data extraction: AxoRouter extracts the relevant information from the content of the messages, and makes it available as structured data.

   The router can perform other processing steps, as configured in the flows that apply to the specific router (see next step).

4. Flow: You can configure flows in the Axoflow Console that Axoflow uses to configure the AxoRouter instances to filter, route, and process the security data. Flows also allow you to automatically remove unneeded or redundant information from the messages, reducing data volume and SIEM and storage costs.

5. Destination: The router sends data to the specified destination in a format optimized for the specific destination.

2 - Flow management

Axoflow uses flows to manage the routing and processing of security data.

2.1 - Flow overview

Axoflow uses flows to manage the routing and processing of security data. A flow applies to one or more AxoRouter instances. The Flows page lists the configured flows, and also highlights if any alerts apply to a flow.

Each flow consists of the following main elements:

• A Router selector that specifies the AxoRouter instances the flow applies to. Multiple flows can apply to a single AxoRouter instance.
• Processing steps that filter and select the messages to process, set/unset message fields, and perform different data transformation and data reduction.
• A Destination where the AxoRouter instances of the flow deliver the data. Destinations can be external destinations (for example, a SIEM), or other AxoRouter instances.

Based on the flows, Axoflow Console automatically generates and deploys the configuration of the AxoRouter instances. Click ⋮ or the name of the flow to display the details of the flow.

Filter flows

To find or display only specific flows, you can use the filter bar.

• Free Text mode searches in the following fields of the flow: Name, Destination, Description.
• AQL mode allows you to search in specific fields of the flows. It also makes more complex filtering possible, using the Equal, Contains, and Match operators. When using AQL mode, Axoflow Console autocompletes the built-in labels and field names, but doesn’t autocomplete custom labels.

Disable flow

You can disable a flow without deleting it if needed by clicking the toggle on the right of the flow name.

2.2 - Create flow

To create a new flow, open the Axoflow Console, then complete the following steps:

1. Select Flows.

2. Select Create New Flow.

3. Enter a name for the flow, for example, my-test-flow.

   

4. In the Router Selector field, enter an expression that matches the router(s) you want to apply the flow. To select a specific router, use a name selector, for example, name = my-axorouter-hostname.

5. Select the Destination where you want to send your data. If you don’t have any destination configured, see Destinations.

   By default, you can select only external destinations. If you want to send data to another AxoRouter, enable the Show all destinations option, and select the connector of the AxoRouter where you want to send the data.

   

6. (Optional) To process the data transferred in the flow, select Add New Processing Step. For details, see Processing steps. For example:

   1. Add a Reduce step to automatically remove redundant and empty fields from your data.
   2. To select which messages are processed by the flow, add a Select Messages step, and enter a filter into the Query field. For example, to select only the messages received from Fortinet FortiGate firewalls, use the meta.vendor = fortinet + meta.product = fortigate query.
   3. Save the processing steps.

   

7. Select Create.

8. The new flow appears in the Flows list.

   

2.3 - Processing steps

Flows can include a list of processing steps to select and transform the messages. All log messages received by the routers of the flow are fed into the pipeline constructed from the processing steps, then the filtered and processed messages are forwarded to the specified destination.

To add processing steps to an existing flow, complete the following steps.

1. Open the Flows page and select the flow you want to modify.

2. Select Process > Add new Processing Step.

3. Select the type of the processing step to add.

   

   The following types of processing steps are available:

   • Select Messages: Filter the messages using a query. Only the matching messages will be processed in subsequent steps.
   • Set Field: Set the value of a field on the message object.
   • Unset Field: Unset the specified field of the message object.
   • Reduce: Automatically remove redundant content from the messages.
   • Regex: Execute a regular expression on the message.
   • FilterX: Execute an arbitrary Filterx snippet.
   • Probe: A measurement point that provides metrics about the throughput of the flow.

4. Configure the processing step as needed for your environment.

5. (Optional) To apply the processing step only to specific messages from the flow, set the Condition field of the step. Conditions act as filters, but apply only to this step. Messages that don’t match the condition will be processed by the subsequent steps.

6. (Optional) If needed, drag-and-drop the step to change its location in the flow.

7. Select Save.

FilterX

Execute an arbitrary FilterX snippet. The following example checks if the meta.host.labels.team field exists, and sets the meta.openobserve variable to a JSON value that contains stream as a key with the value of the meta.host.labels.team field.

Probe

Set a measurement point that provides metrics about the throughput of the flow. Note that you can’t name your probes input and output, as these are reserved.

For details, see Flow metrics.

Reduce

Automatically remove redundant content from the messages.

Regex

Use a regular expression to process the message.

• Input is an AxoSyslog template string, for example, ${MESSAGE} that the regular expression will be matched with.

• Pattern is a Perl Compatible Regular Expression (PCRE). If you want to match the pattern to the whole string, use the ^ and $ anchors at the beginning and end of the pattern.

  You can use https://regex101.com/ with the ECMAScript flavor to validate your regex patterns. (When denoting named capture groups, ?P is not supported, use ?.)

Named or anonymous capture groups denote the parts of the pattern that will be extracted to the Field field. Named capture groups can be specified like (?<group_name>pattern), which will be available as field_name["group_name"]. Anonymous capture groups will be numbered, for example, the second group from (first.*) (bar|BAR) will be available as field_name["2"]. When specifying variables, note the following points:

• If you want to include the results in the outgoing message or in its metadata for other processing steps, you must:

  • Reference an existing variable or field. This field will be updated with the matches.
  • For new variables, use names for the variables that start with the $ character, or
  • declare them explicitly. For details, see the FilterX data model in the AxoSyslog documentation.

• The field will be set to an empty dictionary ({}) when the pattern doesn’t match.

For example, the following processing step parses an unclassified log message from a firewall into a structured format in the log.body field:

• Input: ${RAWMSG}

• Pattern:

  ^(?<priority><\d+>)(?<version>\d+)\s(?<timestamp>[\d\-T:.Z]+)\s(?<hostname>[\w\-.]+)\s(?<appname>[\w\-]+)\s(?<event_id>\d+|\-)\s(?<msgid>\-)\s(?<unused>\-)\s(?<rule_id>[\w\d]+)\s(?<network>\w+)\s(?<action>\w+)\s(?<result>\w+)\s(?<rule_seq>\d+)\s(?<direction>\w+)\s(?<length>\d+)\s(?<protocol>\w+)\s(?<src_ip>\d+\.\d+\.\d+\.\d+)\/(?<src_port>\d+)->(?<dest_ip>\d+\.\d+\.\d+\.\d+)\/(?<dest_port>\d+)\s(?<tcp_flags>\w+)\s(?<policy>[\w\-.]+)
  

• Field: log.body

Select messages

Filter the messages using a query. Only the matching messages will be processed in subsequent steps.

The following example selects messages that have the resource.attributes["service.name"] label set to nginx.

You can also select messages using various other metadata about the connection and the host that sent the data, the connector that received the data, the classification results, and also custom labels, for example:

• a specific connector: meta.connector.name = axorouter-soup
• a type of connector: meta.connector.type = otlp
• a sender IP address: meta.connection.src_ip = 192.168.1.1
• a specific product: meta.product = fortigate (see Vendors for the metadata of a particular product)
• a custom label you’ve added to the host: meta.host.labels.location = eu-west-1

In addition to exact matches, you can use the following operators:

• != not equal (string match)
• =* contains (substring match)
• !*: doesn’t contain
• =~: matches regular expression
• !~: doesn’t match regular expression
• ==~: matches case sensitive regular expression
• !=~: doesn’t match case sensitive regular expression

You can combine multiple expressions using the AND (+) operator. (To delete an expression, hit SHIFT+Backspace.)

Set field

Set specific field of the message. You can use static values, and also dynamic values that were extracted from the message automatically by Axoflow or manually in a previous processing step. If the field doesn’t exist, it’s automatically created. The following example sets the resource.attributes.format field to parsed.

Unset field

Unset a specific field of the message.

2.4 - Flow metrics

The Metrics page of a flow shows the amount of data (in events per second or bytes per second) processed by the flow. By default, the flow provides metrics for the input and the output of the flow. To get more information about the data flow, add Probe steps to the flow. For example, you can add a probe after:

• Select Messages steps to check the amount of data that match the filter, or
• Reduce steps to check the ratio of data saved.

2.5 - Flow analytics

The Analytics page of a flow allows you to analyze the data throughput of the flow using Sankey and Sunburst diagrams. For details on using these analytics, see Analytics.

2.6 - Paths

Create a path

Open the Axoflow Console.

1. Select Topology > + > Path.

   

2. Select your data source in the Source host field.

   

3. Select the target router or aggregator this source is sending its data to in the Target host field, for example, axorouter.

4. Select the Target connector. The connector determines how the destination receives the data (for example, using which protocol or port).

5. Select Create. The new path appears on the Topology page.

   

2.7 - Flow tapping

Flow tapping allows you to sample the data flowing in a Flow and see how it processes the logs. It’s useful to test the flow when you’re configuring or adjusting Processing Steps.

Prerequisites

Flow tapping works only on flows that have been deployed to an AxoRouter instance. This means that if you change a processing step in a flow, you must first Save the changes (or select Save and Test).

If you want to experiment with a flow, we recommend that you clone ( ) the flow and configure it to use the /dev/null destination, then redo adjusting the final processing steps in the original flow.

Tap into a flow

To tap into a Flow, complete the following steps.

1. Open Axoflow Console and select Flows.

2. Select the flow you want to tap into, then select Process > Test.

   

3. Select the router where you want to tap into the flow, then click Start Log Tapping.

   

4. Axoflow Console displays a sample of the logs from the data flow in matched pairs for the input and the output of the flow.

   The input point is after the source connector finished processing the incoming message. For the Syslog (autodetect and classify), this means that the input messages are already classified.

   The output shows the messages right before they are formatted according to the specific destination’s requirements.

   

5. You can open the details of the messages to compare fields, values, and so on. By default, the messages are linked: opening the details of a message in the output automatically opens the details of the message input window, and vice versa (provided the message you clicked exists in both windows).

   

   If you want to compare a message with other messages, click link to disable linking.

6. If the flow contains one or more Probe steps, you can select the probe as the tapping point. That way you can check how the different processing steps modify your messages. You can change the first and the second tapping point by clicking and , respectively.