Amazon S3

Add an Amazon S3 destination to send your security data to an S3 bucket, and look up the message fields it uses.

Amazon S3: Scalable cloud storage service for storing and retrieving any amount of data via object storage.

To add an Amazon S3 destination to Axoflow, complete the following steps.

Prerequisites

  • An existing S3 bucket configured for programmatic access, and the related ACCESS_KEY and SECRET_KEY of a user that can access it. The user needs to have the following permissions:

    • s3:ListBucket
    • s3:ListBucketMultipartUploads
    • s3:AbortMultipartUpload
    • s3:ListMultipartUploadParts
    • s3:PutObject

    If the default encryption of the bucket is SSE-KMS (aws:kms), the AWS role or user also needs the following permissions on the KMS key:

    See a sample IAM policy
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "bucketLevel",
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:ListBucketMultipartUploads",
                    "s3:AbortMultipartUpload",
                    "s3:ListMultipartUploadParts"
                ],
                "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME"
            },
            {
                "Sid": "objectLevel",
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject"
                ],
                "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
            }
        ]
    }
  • To configure Axoflow, you’ll need the bucket name, region (or URL), access key, and the secret key of the bucket.

Steps

  1. Create a new destination.

    1. Open the AxoConsole.
    2. Select Destinations > Add Destination > Destination.
  2. Configure the destination.

    1. Select AWS S3.

    2. Enter a name for the destination.

      Configure the S3 destination

    3. (Optional) Add custom labels to the destination.

    4. Enter the name of the bucket you want to use into the Bucket name field.

    5. Enter the region code of the bucket into the Region field (for example, us-east-1.), or select the Use custom endpoint URL option, and enter the URL of the endpoint into the URL field.

    6. Enter the Access key and the Secret key for the account you want to use.

    7. Enter the Object key (or key name), which uniquely identifies the object in an Amazon S3 bucket, for example: my-logs/${HOSTNAME}/.

      You can use AxoSyslog macros in this field.

    1. (Optional) Set the Object key timestamp. By default, Use custom object key timestamp is enabled, so you can enter a custom template into the Object key timestamp field. To select one of the predefined formats instead (for example, ${R_YEAR}/${R_MONTH}/${R_DAY}), disable Use custom object key timestamp. For details on the available date-related macros, see the AxoSyslog documentation .

    2. Set the maximal size of the S3 object in the Maximum object size field. If an object reaches this size, Axoflow appends an index ("-1", “-2”, …) to the end of the object key and starts a new object after rotation. Default value: 100 MB.

    3. (Optional) If needed, adjust the size of messages written to the S3 object in a batch using the Chunk size field.

    4. (Optional) Select More options to set other options as needed for your environment.

      • Message format: Specifies the message format and normalization. Possible values:

        • Elastic Common Schema (ECS): Normalize the data using the ECS schema.
        • Open Cybersecurity Schema Framework (OCSF): Normalize the data using the OCSF schema.
        • Parsed: Send the parsed data without further normalization.
    5. Select Add.

  3. Create a flow to connect the new destination to an AxoRouter instance.
    1. Select Flows.

    2. Select Add Flow > Flow.

      To create a fallback flow , select Add Flow > Fallback flow.

      Add flow

    3. Enter a name for the flow, for example, my-test-flow.

      Create a flow

    4. In the Router Selector field, enter an expression that matches the routers you want to apply the flow to. To select a specific router, use a name selector, for example, name = my-axorouter-hostname.

      You can use any labels and metadata of the AxoRouter hosts in the Router selectors, for example, the hostname of the AxoRouter, or any custom labels .

      • If you leave the router selector empty, it matches every AxoRouter instance.
      • To select only a specific AxoRouter instance, set the name field to the name of the instance as selector. For example, name = my-axorouter.
      • If you set multiple fields in the selector, the selector will match only AxoRouter instances that match all elements of the selector. (There in an AND relationship between the fields.)
    5. Select the Destination where you want to send your data. If you don’t have any destination configured, you can select Add Destination in the destination section to create a new destination now. For details on the different destinations, see Destinations.

      • If you don’t have any destination configured, see Destinations.
      • If you’ve already created a store , it’s automatically available as a destination. Note that the Router Selector of the flow must match only AxoRouters that have the selected store available, otherwise you’ll get an error message.
      • If you want to send data to another AxoRouter, enable the Show all destinations option, and select the connector of the AxoRouter where you want to send the data.

      The destination list with the AxoRouter connectors shown

    6. (Optional) To process the data transferred in the flow, select Add New Processing Step. For details, see Processing steps. For example:

      1. Add a Classify, a Parse, and a Reduce step, in that order, to automatically remove redundant and empty fields from your data.
      2. To select which messages are processed by the flow, add a Select Messages step, and enter a filter into the AQL Expression field. For example, to select only the messages received from Fortinet FortiGate firewalls, use the meta.vendor = fortinet AND meta.product = fortigate query.

      Example processing steps

    7. Select Add, then confirm with Add. To save the flow without provisioning it to the routers yet, select Add as draft instead.

    8. The new flow appears in the Flows list.

      The new flow

You can use the following message fields to modify messages sent to this destination using processing steps .

Troubleshooting

If the data doesn’t arrive at the destination, see the data doesn’t reach the destination . To find the errors that the destination returns, see the destination rejects the data . For other problems, see Troubleshooting.