Troubleshooting Axoflow

Find the symptom that matches your problem, then complete the checks in order. The checks use the following tools of AxoConsole:

• Log tapping shows samples of the data that a host receives and sends. To see the internal logs of the services on a host, tap the service logs.
• Flow tapping shows how the processing steps of a flow change the data.
• Alerts show the problems that Axoflow detected in your pipeline.
• Host metrics show the history of the connections, the throughput, the queues, and the dropped messages of a host.
• On the AxoRouter host, the axorouter-ctl command shows the logs and the statistics of AxoRouter.

A source doesn’t appear on the Topology page

1. Check that the source sends its data to an AxoRouter instance that’s registered in AxoConsole. A source appears in the Detected list only after AxoRouter receives data from it. To add the source, see onboard a vendor source.
2. Check that the source sends its data to a port and protocol that a connector of the AxoRouter accepts. For the default ports, see the open ports. To receive data on other ports, see AxoRouter connector rules.
3. Check that the firewall of the AxoRouter host allows traffic on the port of the connector.
4. Check that you configured the source as its page in the Vendors section describes. Other message formats or ports can cause data loss or incorrect parsing.
5. Tap the service logs of the AxoRouter, and look for errors about the connector, for example, TLS handshake errors.

AxoRouter receives the data, but it doesn’t reach the destination

1. Check that a flow sends the data to the destination:

   • The Router Selector of the flow matches the AxoRouter instance that receives the data.
   • The flow is enabled.

   For details, see Flow overview.

2. Check that the processing steps of the flow don’t filter out the data, for example, a Select Messages step. To find the step that drops the data, add Probe steps to the flow, and check the flow metrics, or use Flow tapping.

3. If no other flow processes the data, check whether a fallback flow sends it to a different destination.

4. Check that the destination accepts the data. See The destination rejects the data.

The destination rejects the data

Check the logs of the AxoRouter that sends data to the destination:

1. Select Topology, then select your AxoRouter instance. Alternatively, select ⌘/Ctrl + K and enter the name of your AxoRouter.

2. Select ⋮ Host Options > Tap service logs.

3. (Optional) To tap only the logs of a specific service, set the Service name field.

   Depending on the configuration of the host, the following services can be available:

   • axoflow-cloudconnector: The service that collects logs from cloud services and APIs.
   • axoflow-otel-collector: The collector agent on edge hosts.
   • axorouter-logstore: The service that handles Logstore type stores.
   • axolet: The monitoring and management agent for Axoflow pipeline elements.
   • axorouter-resolver: A service for retrieving host data from external CMDB systems.
   • axorouter-syslog: The main processing element of AxoRouter deployments.
   • axorouter-wec: The service that handles the Windows Events connector (WEC).
   • axostore: The service that handles AxoStore on the host. Only available if there are stores configured on the host.

4. If systemd is available on the host, select Systemd service output to show the logs of the service from the systemd journal. Otherwise, the Internal logs of the service are available.

5. Select Start. Axoflow displays the log messages of AxoRouter. Check the logs for error messages.

The following sections list common errors of specific destinations.

Splunk

Some common errors include:

• Redirected event for unconfigured/disabled/deleted index=netops with source="source::axo" host="host::axosyslog-almalinux" sourcetype="sourcetype::fortigate_event" into the LastChanceIndex. So far received events from 1 missing index(es).: The Splunk index where AxoRouter is trying to send data doesn’t exist. Check which index is missing in the error message and create it in Splunk. (For a list of recommended indices, see the Splunk destination prerequisites.)
• http: error sending HTTP request; url='https://prd-p-sp2id.splunkcloud.com:8088/services/collector/event/1.0?index=&source=&sourcetype=', error='SSL peer certificate or SSH remote key was not OK', worker_index='0', driver='splunk--flow-axorouter4-almalinux#0', location='/usr/share/syslog-ng/include/scl/splunk/splunk.conf:104:3': Your Splunk deployment uses an invalid or self-signed certificate, and the Verify server certificate option is enabled in the Splunk destination of Axoflow. Either fix the certificate in Splunk, or: select Topology > <your-splunk-destination>, disable Verify server certificate, then select Update.

Google Security Operations (SecOps)

If your logs aren’t showing up in Google SecOps, tap the service logs of the axorouter-syslog service on an AxoRouter that’s sending data to Google SecOps.

• If you see errors like http: Server returned with a 4XX (client errors) status code, which means we are not authorized or the URL is not found;, double-check the Service endpoint (make sure that it points to the right regional service endpoint) and Customer ID fields.

AxoRouter drops messages

1. Check the alerts of the AxoRouter host. Axoflow raises alerts when a host drops packets or messages, and when its disk queues fill up.

2. Check the Dropped packets (total) chart and the queue charts in the metrics of the host:

   • Dropped UDP packets means that the operating system dropped UDP packets before AxoRouter could process them.
   • Dropped log events means that the event queues of AxoRouter dropped messages.
   • Growing memory and disk queues mean that a destination doesn’t accept the data as fast as AxoRouter receives it. Check the errors of the destination. See The destination rejects the data.

3. If the host drops UDP packets, increase the throughput of the connector that receives them. UDP has no flow control: when the packets of a connector arrive faster than AxoRouter can process them, for example, during a burst of traffic from a firewall, the operating system drops the packets that don’t fit into the receive buffer. This can happen even if the host has free CPU capacity, because by default a single thread receives and processes the messages of a connector. A larger receive buffer absorbs short bursts, but doesn’t help when the sustained rate is higher than what a single thread can process.

   Edit the connector:

   • Check the Socket buffer size of the connector and the net.core.rmem_max sysctl on the host. The recommended value is 32 MiB.
   • Set the number of Processing workers to process the messages of the connector on multiple threads.
   • If UDP socket count is higher than 1, enable EBPF load balancing. Otherwise, every message from a sender goes to the same socket, so a single high-volume sender doesn’t benefit from the additional sockets. For details, see UDP load balancing.
   • If the source can send its data over TCP, use a TCP connector instead. Over TCP, AxoRouter slows the sender down instead of dropping data.

Disk is full on AxoRouter, or a service fills the disk with logs

1. Check what uses the space. Run df -h /var /run, journalctl --disk-usage, and sudo du -xsh /var/log/* | sort -h on the host.
2. If the logs use the space, find the service that writes them, free disk space, and limit the logs of the service. For details, see A service fills the disk with logs.
3. If the logs don’t use the space, check the disk queue charts in the metrics of the host. Growing disk queues mean that a destination doesn’t accept the data as fast as AxoRouter receives it. See The destination rejects the data.

An edge host doesn’t send data

1. Check that a collection rule applies to the host: the Edge selector of the rule must match the labels of the host. For details, see add an edge host to a collection rule.
2. Check that a data forwarding rule sends the data of the host to an AxoRouter instance.
3. Check that the axolet and axoflow-otel-collector services are running on the host. For details, see Linux or Windows.

A host is unavailable, or its configuration doesn’t update

1. Check the alerts of the host. Axoflow raises alerts when a host becomes unavailable, and when the configuration of a managed host fails to synchronize or reload.
2. Check that the axolet service is running on the host. For details, see Linux or Windows.
3. If you cloned the host from another host that runs Axolet, or the root CA certificate of AxoConsole changed, reinstall Axolet. For details, see Reinstall Axolet.

Installation problems

AxoConsole on premises

• If you get the [ERROR] Timed out waiting for k3s node to be Ready error message, increase the timeout limit by running K3S_TIMEOUT=180, then rerun the installation script.

In case of other errors, run the following command to create a debug package from the cluster, and contact our support team.

sudo k3s kubectl cluster-info dump --all-namespaces  --output-directory=./cluster-dump && tar -czvf cluster-dump.tar.gz cluster-dump

Axoflow agent for Windows

If you run into problems when installing Axoflow agent for Windows, complete the following steps.

1. Double-check that you’re running the installation command:

   • as administrator,
   • in PowerShell, not in Cmd.

2. Set the following environment variable to store the installation logs. Adjust the directory as needed.

   $env:TEMP_DIR = "C:\Temp\axoflow-agent-install"

3. Re-run the installation command.

4. Open the Windows Event Viewer and check the Windows Logs > System events at the time of installation for errors or issues logged by the Service Control Manager.

5. Check that the installation logs of axolet and axoflow-otel-collector are available, for example, the C:\Temp\axoflow-agent-install\axolet-windows-amd64.msi-install.log and C:\Temp\axoflow-agent-install\axoflow-otel-collector_<version-number>-axoflow.1_windows_x64.msi-install.log files exist.

6. If you suspect that the installer is incorrectly reporting connectivity issues, disable the preflight check and re-run the installation command.

   $env:SKIP_PREFLIGHT_CHECK = 'true'

7. If needed, contact our support team.

Contact support

If you can’t solve the problem, contact the Axoflow support team.