Amazon S3

To add an Amazon S3 destination to Axoflow, complete the following steps.

Prerequisites

  • An existing S3 bucket configured for programmatic access, and the related ACCESS_KEY and SECRET_KEY of a user that can access it. The user needs to have the following permissions:

    • kms:Decrypt
    • kms:Encrypt
    • kms:GenerateDataKey
    • s3:ListBucket
    • s3:ListBucketMultipartUploads
    • s3:AbortMultipartUpload
    • s3:ListMultipartUploadParts
    • s3:PutObject

  • To configure Axoflow, you’ll need the bucket name, region (or URL), access key, and the secret key of the bucket.

Steps

  1. Create a new destination.

    1. Open the Axoflow Console.
    2. Select Topology.
    3. Select + > Destination.

  2. Configure the destination.

    1. Select Amazon S3.

    2. Enter a name for the destination.

      Configure the S3 destination

    3. Enter the name of the bucket you want to use.

    4. Enter the region code of the bucket into the Region field (for example, us-east-1.), or select the Use custom endpoint URL option, and enter the URL of the endpoint into the URL field.

    5. Enter the Access key and the Secret key for the account you want to use.

    6. Enter the Object key (or key name), which uniquely identifies the object in an Amazon S3 bucket, for example: my-logs/${HOSTNAME}/.

      You can use AxoSyslog macros in this field.

    1. Select the Object key timestamp format you want to use, or select Use custom object key timestamp and enter a custom template. For details on the available date-related macros, see the AxoSyslog documentation.
    2. Set the maximal size of the S3 object. If an object reaches this size, Axoflow appends an index ("-1", “-2”, …) to the end of the object key and starts a new object after rotation.
    3. Select Create.
  3. Create a flow to connect the new destination to an AxoRouter instance.

    1. Select Flows.

    2. Select Create New Flow.

    3. Enter a name for the flow, for example, my-test-flow.

      Create a flow

    4. In the Router Selector field, enter an expression that matches the router(s) you want to apply the flow. To select a specific router, use a name selector, for example, name = my-axorouter-hostname.

    5. Select the Destination where you want to send your data. If you don’t have any destination configured, see Destinations.

    6. (Optional) To process the data transferred in the flow, select Add New Processing Step. For details, see Processing steps. For example:

      1. Add a Reduce step to automatically remove redundant and empty fields from your data.
      2. To select which messages are processed by the flow, add a Select Messages step, and enter a filter into the Query field. For example, to select only the messages received from Fortinet Fortigate firewalls, use the meta.vendor = fortinet + meta.product = fortigate query.
      3. Save the processing steps.

      Example processing steps

    7. Select Create.

    8. The new flow appears in the Flows list.

      The new flow