Google

• 1: Google Cloud Pub/Sub

1 - Google Cloud Pub/Sub

To add a Google Cloud Pub/Sub destination to Axoflow, complete the following steps.

Prerequisites

• A Google Pub/Sub subscription.
• An IAM service account that Axoflow uses for authentication.
• A Google Cloud project that has the Pub/Sub API enabled.

For details, see the Google Pub/Sub tutorial.

Steps

1. Create a new destination.

   1. Open the Axoflow Console.
   2. Select Topology.
   3. Select + > Destination.

2. Configure the destination.

   1. Select Pub/Sub.

   2. Select which type of configuration you want to use:

      • Simple: Send all data into a project and topic.
      • Dynamic: Send data to a project and topic based on the content or metadata of the incoming messages (or to a default project/topic).

   3. Enter a name for the destination.

      

   4. Specify the project and topic to send the data to.

      • Simple: Enter the ID of the GCP Project and the Topic. All data will be sent to this topic.
      • Dynamic: Enter the expression that specifies the default Project and Topic. The data will be sent into here unless it is set during the processing of the message (for example, by the processing steps of the Flow).

      You can use AxoSyslog macros in this field.

   5. Configure the authentication method to access the GCP project.

      • Automatic (ADC): Use the service account attached to the cloud resource (VM) that hosts AxoRouter.

      • Service Account File: Specify the path where a service account key file is located (for example, /etc/axorouter/user-config/). You must manually copy that file to its place, currently you can’t distribute it from Axoflow.

      • None: Disable authentication completely. Only available when the More options > Service Endpoint option is set.

        CAUTION:

        Do not disable authentication in production.

   6. (Optional) Set other options as needed for your environments.

      • Service Endpoint: Use a custom API endpoint. Leave it empty to use the default: https://pubsub.googleapis.com
      • Timeout: The number of seconds to wait for a log-forwarding request to complete, and attempt to reconnect the server if exceeded. The default (0) is unlimited. For more details, see the AxoSyslog documentation.
      • Batch Timeout: Maximal time in milliseconds to wait for a batch to be filled before sending it. The default value (-1) means that the batch should be full before sending, which may result in a long wait if the incoming traffic is low. For more details, see the AxoSyslog documentation.
      • Workers: Used for scaling the destination in case of high message load. Specifies the number of worker threads AxoRouter uses for sending messages to the destination. The default is 1. If high message throughput is not a concern, leave it on 1. For maximum performance, increase it up to the number of CPU cores available on AxoRouter. For more details, see the AxoSyslog documentation.

   7. Select Create.

3. Create a flow to connect the new destination to an AxoRouter instance.

   1. Select Flows.

   2. Select Create New Flow.

   3. Enter a name for the flow, for example, my-test-flow.

      

   4. In the Router Selector field, enter an expression that matches the router(s) you want to apply the flow. To select a specific router, use a name selector, for example, name = my-axorouter-hostname.

   5. Select the Destination where you want to send your data. If you don’t have any destination configured, see Destinations.

      By default, you can select only external destinations. If you want to send data to another AxoRouter, enable the Show all destinations option, and select the connector of the AxoRouter where you want to send the data.

      

   6. (Optional) To process the data transferred in the flow, select Add New Processing Step. For details, see Processing steps. For example:

      1. Add a Reduce step to automatically remove redundant and empty fields from your data.
      2. To select which messages are processed by the flow, add a Select Messages step, and enter a filter into the Query field. For example, to select only the messages received from Fortinet FortiGate firewalls, use the meta.vendor = fortinet + meta.product = fortigate query.
      3. Save the processing steps.

      

   7. Select Create.

   8. The new flow appears in the Flows list.

      

Pub/Sub attributes

You can embed custom attributes as metadata in Pub/Sub messages in the processing steps of the Flow that’s sending data to the Pub/Sub destination.

To do that:

1. Add a FilterX processing step to the Flow.

2. Edit the Statements field of the processing step:

   1. Add the meta.pubsub.attributes = json(); line to add an empty JSON object to the messages.

   2. Set your custom attributes under the meta.pubsub.attributes key. For example, if you want to include the timestamp as a custom attribute as well, you can use:

      meta.pubsub.attributes = {"timestamp": $S_ISODATE};