Amazon S3

Amazon S3: Scalable cloud storage service for storing and retrieving any amount of data via object storage.

To add an Amazon S3 destination to Axoflow, complete the following steps.

Prerequisites

  • An existing S3 bucket configured for programmatic access, and the related ACCESS_KEY and SECRET_KEY of a user that can access it. The user needs to have the following permissions:

    • kms:Decrypt
    • kms:Encrypt
    • kms:GenerateDataKey
    • s3:ListBucket
    • s3:ListBucketMultipartUploads
    • s3:AbortMultipartUpload
    • s3:ListMultipartUploadParts
    • s3:PutObject

  • To configure Axoflow, you’ll need the bucket name, region (or URL), access key, and the secret key of the bucket.

Steps

  1. Create a new destination.

    1. Open the AxoConsole.
    2. Select Destinations > + Add Destination.

  2. Configure the destination.

    1. Select Amazon S3.

    2. Enter a name for the destination.

      Configure the S3 destination

    3. Enter the name of the bucket you want to use.

    4. Enter the region code of the bucket into the Region field (for example, us-east-1.), or select the Use custom endpoint URL option, and enter the URL of the endpoint into the URL field.

    5. Enter the Access key and the Secret key for the account you want to use.

    6. Enter the Object key (or key name), which uniquely identifies the object in an Amazon S3 bucket, for example: my-logs/${HOSTNAME}/.

      You can use AxoSyslog macros in this field.

    1. Select the Object key timestamp format you want to use, or select Use custom object key timestamp and enter a custom template. For details on the available date-related macros, see the AxoSyslog documentation.
    2. Set the maximal size of the S3 object. If an object reaches this size, Axoflow appends an index ("-1", “-2”, …) to the end of the object key and starts a new object after rotation.
    3. Select Add.
  3. Create a flow to connect the new destination to an AxoRouter instance.

    1. Select Flows.

    2. Select Add Flow.

    3. Enter a name for the flow, for example, my-test-flow.

      Create a flow

    4. In the Router Selector field, enter an expression that matches the router(s) you want to apply the flow. To select a specific router, use a name selector, for example, name = my-axorouter-hostname.

      You can use any labels and metadata of the AxoRouter hosts in the Router selectors, for example, the hostname of the AxoRouter, or any custom labels.

      • If you leave the Router Selector field empty, the selector will match every AxoRouter instance.
      • To select only a specific AxoRouter instance, set the name field to the name of the instance as selector. For example, name = my-axorouter.
      • If you set multiple fields in the selector, the selector will match only AxoRouter instances that match all elements of the selector. (There in an AND relationship between the fields.)

    5. Select the Destination where you want to send your data. If you don’t have any destination configured, you can select + Add in the destination section to create a new destination now. For details on the different destinations, see Destinations.

      • If you don’t have any destination configured, see Destinations.
      • If you’ve already created a store, it automatically available as a destination. Note that the Router Selector of the flow must match only AxoRouters that have the selected store available, otherwise you’ll get an error message.
      • If you want to send data to another AxoRouter, enable the Show all destinations option, and select the connector of the AxoRouter where you want to send the data.

      AxoRouter as destination

    6. (Optional) To process the data transferred in the flow, select Add New Processing Step. For details, see Processing steps. For example:

      1. Add a Classify, a Parse, and a Reduce step, in that order, to automatically remove redundant and empty fields from your data.
      2. To select which messages are processed by the flow, add a Select Messages step, and enter a filter into the AQL Expression field. For example, to select only the messages received from Fortinet FortiGate firewalls, use the meta.vendor = fortinet AND meta.product = fortigate query.
      3. Save the processing steps.

      Example processing steps

    7. Select Add.

    8. The new flow appears in the Flows list.

      The new flow

You can use the following message fields to modify messages sent to this destination using processing steps.