Splunk HEC Server

Splunk HEC Server connectors of AxoRouter can receive log events from Splunk forwarders and any other client that posts to the Splunk HTTP Event Collector (HEC), for example, Splunk Edge Processor, Splunk OpenTelemetry Collector, or Splunk Connect for Syslog (SC4S). Point such a client at AxoRouter instead of your Splunk HEC endpoint, and Axoflow ingests the events it sends.

Prerequisites

To receive data over HTTPS, you’ll need a key and a certificate that the connector shows to the clients.

If you want to enable TLS encryption for this connector to encrypt the communication with the sources, you’ll need to set appropriate keys and certificates.

Note the following points:

• Keys and certificates must be in PEM format.
• If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.
• If you manually copy these files to their place on the AxoRouter host, make the files readable by the axorouter service.
  • The recommended path for certificates is under /etc/axorouter/user-config/ (for example, /etc/axorouter/user-config/tls-key.pem). (If you need to use a different path, you have to append an option like -v /your/path:/your/path to the AXOROUTER_PODMAN_ARGS variable of /etc/axorouter/container.env.)
  • When referring to the key or certificate when configuring the connector, use absolute paths (for example, /etc/axorouter/user-config/tls-key.pem).

Add new Splunk HEC Server connector

To add a new connector to an AxoRouter host, complete the following steps:

1. Select Routers > Connector Rules > Add Connector Rule. (Alternatively, you can select Add Connector > Add a connector rule on the Connectors page of an AxoRouter host.)

   

2. Select Splunk HEC Server.

3. Configure the connector rule.

   1. Enter a name for the connector rule into the Connector rule name field.

      

   2. (Optional) Add labels to the connector rule. You will be able to use these labels in Flow Processing steps, for example, in the Query field of Select Messages steps. For details about the message schema and the related connector-specific fields, see the meta.connector object.

   3. Set the Router selector for the connector rule. The selector determines which AxoRouter instances will have a connector based on this connector rule.

      

      You can use any labels and metadata of the AxoRouter hosts in the Router selectors, for example, the hostname of the AxoRouter, or any custom labels.

      • If you leave the router selector empty, it matches every AxoRouter instance.
      • To select only a specific AxoRouter instance, set the name field to the name of the instance as selector. For example, name = my-axorouter.
      • If you set multiple fields in the selector, the selector will match only AxoRouter instances that match all elements of the selector. (There in an AND relationship between the fields.)

   4. (Optional) Enter a Suffix for the connector rule. AxoConsole uses the suffix in the name of the connector instances that it creates on the AxoRouter hosts. The name of the connector instance follows the <name-of-AxoRouter>-<suffix> schema. For example, if the name of a matching AxoRouter instance is axorouter-1, and the suffix of the rule is custom, the name of the connector on that AxoRouter is axorouter-1-custom.

      If the Suffix field is empty, AxoConsole uses the name of the connector rule instead. For example, if the name of the connector rule is otlp-1, the name of the connector is axorouter-1-otlp-1.

   5. (Optional) Enter a description for the rule.

   6. If needed, turn on the Classify and Parse steps in the Preprocessing steps section (they’re off by default), so AxoRouter automatically identifies and parses messages sent by supported data sources. If your source is not listed, contact us.

      Enabling these options processes all data received by the connectors created based on this connector rule. If you want to apply classification and parsing more selectively, you can use the Classify and Parse processing steps in your Flows.

      Note that the Parse processing step requires Classify to be enabled. Parsing automatically parses the data from the content of the message, and replaces the message content (the log.body field in the internal message schema) with the structured information.

   7. (Optional) If needed, you can add other preprocessing steps to the connector rule. You can use the same processing steps as in flows. These can be useful if you use a dedicated connector rule to fix or annotate data coming from some specific sources. To reuse the preprocessing steps in other connector rules or flows, select Save as Custom Step.

4. Enter the HEC Token that the clients must present. This option is required, the connector has no unauthenticated mode.

   The clients send this token in the Authorization: Splunk <token> header of every request, and AxoRouter refuses the requests that don’t carry it.

5. (Optional) Set the Port number where the connector receives the requests. If you leave it empty, the connector listens on port 8088 by default.

   CAUTION:

   TCP and UDP port range 29500–29519 is reserved for AxoRouter’s internal listeners and cannot be used in connector rules.

6. (Optional) To receive data over HTTPS, select Enable TLS, then set the certificate and the private key that AxoRouter shows to the clients.

   Note These settings apply to every AxoRouter that matches the Router selector of the connector rule. If you paste values for the server certificate and server private key, every matching AxoRouter uses the same shared certificate, which is not recommended. Instead, use a separate certificate for each AxoRouter by setting the file path option, or create a separate connector rule for each AxoRouter.

   • Server certificate: The certificate that AxoRouter shows to the clients.

     • To provide the server certificate from the AxoConsole, select Paste value and paste the server certificate. AxoConsole will automatically deploy the server certificate to the related AxoRouters.
     • To provide the server certificate from a file, select File path and enter the path where the certificate file is located (for example, /etc/axorouter/user-config/server-certificate.pem). The path must be mounted into the container, /etc/axorouter is mounted by default. You must manually copy that file to its place on every affected AxoRouter, or you can use the configuration management system of your choice.

   • Server private key: The private key of the server certificate.

     • To provide the server private key from the AxoConsole, select Paste value and paste the server private key. AxoConsole will automatically deploy the server private key to the related AxoRouters.
     • To provide the server private key from a file, select File path and enter the path where the private key file is located (for example, /etc/axorouter/user-config/server-private-key.pem). The path must be mounted into the container, /etc/axorouter is mounted by default. You must manually copy that file to its place on every affected AxoRouter, or you can use the configuration management system of your choice.

7. (Optional) Select More options to change the advanced options of the connector.

8. Select Add.

   Axoflow automatically creates connectors on the AxoRouter hosts that match the Router selector.

   Make sure to enable the ports you’ve configured in the connector on the firewall of the AxoRouter host, and on other firewalls between the AxoRouter host and your data sources.

9. Send a test request to the connector.

   1. Open the Overview tab of the AxoRouter host where you’ve created the connector and check the IP address or FQDN of the host.

   2. Open a terminal on your computer and send an event to the /services/collector endpoint of the connector.

      curl -X POST -H 'Authorization: Splunk <your-token>' --data '{"host":"myhost","event":"sample HEC message"}' http://<router-IP-address>:<connector-port-number>/services/collector

10. Configure your clients to send their data to AxoRouter. Set the HEC endpoint of the client to the address and the port of the connector, and the HEC token to the Token of the connector.

Advanced options

Select More options on the connector rule to set the following options.

• Maximum connections: The maximum number of connections that the connector accepts at the same time.
• Maximum request size: The maximum accepted size of a request body, in bytes. AxoRouter refuses the requests that are larger than this.

You can also adjust the flow-control related options.

• Log window size: The size (number of messages) of the initial window used in flow control. For details, see the AxoSyslog documentation.

  If you change the value of Log window size, the change will affect only new connections, the Log window size of already established connections will not change. To apply the new Log window size value to every connection, restart the axorouter.service service on the Services page of the matching AxoRouters. A simple configuration reload is NOT sufficient.

  If the connector rule is receiving data using the UDP protocol, always restart the axorouter.service service on the Services page of the matching AxoRouters.

• Log fetch limit: The maximum number of messages fetched from a source during a single poll loop. For details, see the AxoSyslog documentation.

Metadata fields

AxoRouter unwraps the HEC envelope of every request and turns each event into a separate message. It adds the following fields to the meta variable:

AxoRouter reads the following fields of the HEC envelope as well:

• The event field becomes log.body. The rest of the envelope doesn’t appear in the body of the message.
• The host field becomes meta.host.name.
• The time field becomes the message timestamp (epoch seconds, fractional part allowed) log.time_unix_nano.
• Other fields are dropped.

You can add further parsing to the body using processing steps in Flows, for example, using FilterX or Regex processing steps.

You can use the labels as:

• labels in Customize Graph on the Analytics page,
• in the Filter labels field during log tapping.

You can use the message fields

• in Flow Processing steps, for example, in the Query field of Select Messages steps,
• in AQL expressions in the search bars.