---
title: "Troubleshooting Axoflow"
url: "https://axoflow.com/docs/axoflow/troubleshoot/"
description: "Find the cause of a problem from what you see, like a source that doesn't show up or data that doesn't reach the destination."
last_modified: "2026-10-27T09:35:49+01:00"
---

> For the complete documentation index, see [llms.txt](https://axoflow.com/docs/axoflow/llms.txt).

# Troubleshooting Axoflow

Find the cause of a problem from what you see, like a source that doesn’t show up or data that doesn’t reach the destination.

Find the symptom that matches your problem, then complete the checks in order. The checks use the following tools of AxoConsole:

- [Log tapping](https://axoflow.com/docs/axoflow/onboard-hosts/log-tapping/index.md) shows samples of the data that a host receives and sends. To see the internal logs of the services on a host, [tap the service logs](https://axoflow.com/docs/axoflow/onboard-hosts/log-tapping/index.md#service-tap).
- [Flow tapping](https://axoflow.com/docs/axoflow/data-management/flow-tapping/index.md) shows how the processing steps of a flow change the data.
- [Alerts](https://axoflow.com/docs/axoflow/onboard-hosts/alerts/index.md) show the problems that Axoflow detected in your pipeline.
- [Host metrics](https://axoflow.com/docs/axoflow/metrics/host-metrics/index.md) show the history of the connections, the throughput, the queues, and the dropped messages of a host.
- On the [AxoRouter](https://axoflow.com/docs/axoflow/architecture/index.md#router) host, the [`axorouter-ctl`](https://axoflow.com/docs/axoflow/reference/man-page/axorouter-ctl/index.md) command shows the logs and the statistics of AxoRouter.

## A source doesn’t appear on the Topology page

1. Check that the source sends its data to an AxoRouter instance that’s registered in AxoConsole. A source appears in the **Detected** list only after AxoRouter receives data from it. To add the source, see [onboard a vendor source](https://axoflow.com/docs/axoflow/data-sources/appliances/index.md).
2. Check that the source sends its data to a port and protocol that a connector of the AxoRouter accepts. For the default ports, see the [open ports](https://axoflow.com/docs/axoflow/data-sources/generic/index.md#open-ports). To receive data on other ports, see [AxoRouter connector rules](https://axoflow.com/docs/axoflow/data-sources/connector-rules/index.md).
3. Check that the firewall of the AxoRouter host allows traffic on the port of the connector.
4. Check that you configured the source as its page in the [Vendors section](https://axoflow.com/docs/axoflow/data-sources/appliances/index.md) describes. Other message formats or ports can cause data loss or incorrect parsing.
5. [Tap the service logs](https://axoflow.com/docs/axoflow/onboard-hosts/log-tapping/index.md#service-tap) of the AxoRouter, and look for errors about the connector, for example, TLS handshake errors.

## AxoRouter receives the data, but it doesn’t reach the destination

1. Check that a flow sends the data to the destination:

   - The **Router Selector** of the flow matches the AxoRouter instance that receives the data.
   - The flow is enabled.

   For details, see [Flow overview](https://axoflow.com/docs/axoflow/data-management/flow-overview/index.md).
2. Check that the processing steps of the flow don’t filter out the data, for example, a **Select Messages** step. To find the step that drops the data, add **Probe** steps to the flow, and check the [flow metrics](https://axoflow.com/docs/axoflow/data-management/metrics/index.md), or use [Flow tapping](https://axoflow.com/docs/axoflow/data-management/flow-tapping/index.md).
3. If no other flow processes the data, check whether a [fallback flow](https://axoflow.com/docs/axoflow/data-management/flow-overview/index.md#fallback-flow) sends it to a different destination.
4. Check that the destination accepts the data. See [The destination rejects the data](https://axoflow.com/docs/axoflow/troubleshoot/index.md#destination-errors).

## The destination rejects the data

Check the logs of the AxoRouter that sends data to the destination:

1. Select **Topology**, then select your AxoRouter instance. Alternatively, select [`⌘/Ctrl + K`](https://axoflow.com/docs/axoflow/reference/global-search/index.md) and enter the name of your AxoRouter.
2. Select **⋮ Host Options > Tap service logs**.
3. (Optional) To tap only the logs of a specific service, set the **Service name** field.

   Depending on the configuration of the host, the following services can be available:

   - **axoflow-cloudconnector**: The service that collects logs from cloud services and APIs.
   - **axoflow-otel-collector**: The collector agent on edge hosts.
   - **axorouter-logstore**: The service that handles Logstore type stores.
   - **axolet**: The monitoring and management agent for Axoflow pipeline elements.
   - **axorouter-resolver**: A service for retrieving host data from external CMDB systems.
   - **axorouter-syslog**: The main processing element of AxoRouter deployments.
   - **axorouter-wec**: The service that handles the [Windows Events connector (WEC)](https://axoflow.com/docs/axoflow/data-sources/wec/index.md).
   - **axostore**: The service that handles AxoStore on the host. Only available if there are [stores](https://axoflow.com/docs/axoflow/storage/index.md) configured on the host.
4. If systemd is available on the host, select **Systemd service output** to show the logs of the service from the systemd journal. Otherwise, the **Internal logs** of the service are available.
5. Select **Start**. Axoflow displays the log messages of AxoRouter. Check the logs for error messages.

The following sections list common errors of specific destinations.

### Splunk

Some common errors include:

- `Redirected event for unconfigured/disabled/deleted index=netops with source="source::axo" host="host::axosyslog-almalinux" sourcetype="sourcetype::fortigate_event" into the LastChanceIndex. So far received events from 1 missing index(es).`: The Splunk index where AxoRouter is trying to send data doesn’t exist. Check which index is missing in the error message and create it in Splunk. (For a list of recommended indices, see the [Splunk destination prerequisites](https://axoflow.com/docs/axoflow/destinations/splunk/index.md).)
- `http: error sending HTTP request; url='https://prd-p-sp2id.splunkcloud.com:8088/services/collector/event/1.0?index=&source=&sourcetype=', error='SSL peer certificate or SSH remote key was not OK', worker_index='0', driver='splunk--flow-axorouter4-almalinux#0', location='/usr/share/syslog-ng/include/scl/splunk/splunk.conf:104:3'`: Your Splunk deployment uses an invalid or self-signed certificate, and the **Verify server certificate** option is enabled in the Splunk destination of Axoflow. Either fix the certificate in Splunk, or: select **Topology > <your-splunk-destination>**, disable **Verify server certificate**, then select **Update**.

### Google Security Operations (SecOps)

If your logs aren’t showing up in Google SecOps, [tap the service logs of the axorouter-syslog service](https://axoflow.com/docs/axoflow/onboard-hosts/log-tapping/index.md#service-tap) on an AxoRouter that’s sending data to Google SecOps.

- If you see errors like `http: Server returned with a 4XX (client errors) status code, which means we are not authorized or the URL is not found;`, double-check the **Service endpoint** (make sure that it points to the right [regional service endpoint](https://cloud.google.com/chronicle/docs/reference/ingestion-api#regional_endpoints)) and **Customer ID** fields.

## AxoRouter drops messages

1. Check the [alerts](https://axoflow.com/docs/axoflow/onboard-hosts/alerts/index.md) of the AxoRouter host. Axoflow raises alerts when a host drops packets or messages, and when its disk queues fill up.
2. Check the **Dropped packets (total)** chart and the queue charts in the [metrics of the host](https://axoflow.com/docs/axoflow/metrics/host-metrics/index.md):

   - **Dropped UDP packets** means that the operating system dropped UDP packets before AxoRouter could process them.
   - **Dropped log events** means that the event queues of AxoRouter dropped messages.
   - Growing memory and disk queues mean that a destination doesn’t accept the data as fast as AxoRouter receives it. Check the errors of the destination. See [The destination rejects the data](https://axoflow.com/docs/axoflow/troubleshoot/index.md#destination-errors).
3. If the host drops UDP packets, increase the throughput of the connector that receives them. UDP has no flow control: when the packets of a connector arrive faster than AxoRouter can process them, for example, during a burst of traffic from a firewall, the operating system drops the packets that don’t fit into the receive buffer. This can happen even if the host has free CPU capacity, because by default a single thread receives and processes the messages of a connector. A larger receive buffer absorbs short bursts, but doesn’t help when the sustained rate is higher than what a single thread can process.

   Edit the connector:

   - Check the **Socket buffer size** of the connector and the `net.core.rmem_max` sysctl on the host. The recommended value is 32 MiB.
   - Set the number of [**Processing workers**](https://axoflow.com/docs/axoflow/data-sources/syslog/index.md#processing-workers) to process the messages of the connector on multiple threads.
   - If **UDP socket count** is higher than 1, enable **EBPF load balancing**. Otherwise, every message from a sender goes to the same socket, so a single high-volume sender doesn’t benefit from the additional sockets. For details, see [UDP load balancing](https://axoflow.com/docs/axoflow/data-sources/syslog/index.md#udp-loadbalancing).
   - If the source can send its data over TCP, use a TCP connector instead. Over TCP, AxoRouter slows the sender down instead of dropping data.

## Disk is full on AxoRouter, or a service fills the disk with logs

1. Check what uses the space. Run `df -h /var /run`, `journalctl --disk-usage`, and `sudo du -xsh /var/log/* | sort -h` on the host.
2. If the logs use the space, find the service that writes them, free disk space, and limit the logs of the service. For details, see [A service fills the disk with logs](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md).
3. If the logs don’t use the space, check the disk queue charts in the [metrics of the host](https://axoflow.com/docs/axoflow/metrics/host-metrics/index.md). Growing disk queues mean that a destination doesn’t accept the data as fast as AxoRouter receives it. See [The destination rejects the data](https://axoflow.com/docs/axoflow/troubleshoot/index.md#destination-errors).

## An edge host doesn’t send data

1. Check that a [collection rule](https://axoflow.com/docs/axoflow/data-sources/collection-rules/index.md) applies to the host: the **Edge selector** of the rule must match the labels of the host. For details, see [add an edge host to a collection rule](https://axoflow.com/docs/axoflow/data-sources/collection-rules/index.md#add).
2. Check that a [data forwarding rule](https://axoflow.com/docs/axoflow/data-sources/data-forwarding/index.md) sends the data of the host to an AxoRouter instance.
3. Check that the `axolet` and `axoflow-otel-collector` services are running on the host. For details, see [Linux](https://axoflow.com/docs/axoflow/provisioning/linux-agent/manage/index.md#check-status) or [Windows](https://axoflow.com/docs/axoflow/provisioning/windows-opentelemetry/manage-windows-agent/index.md#check-status).

## A host is unavailable, or its configuration doesn’t update

1. Check the [alerts](https://axoflow.com/docs/axoflow/onboard-hosts/alerts/index.md) of the host. Axoflow raises alerts when a host becomes unavailable, and when the configuration of a managed host fails to synchronize or reload.
2. Check that the `axolet` service is running on the host. For details, see [Linux](https://axoflow.com/docs/axoflow/provisioning/linux-agent/manage/index.md#check-status) or [Windows](https://axoflow.com/docs/axoflow/provisioning/windows-opentelemetry/manage-windows-agent/index.md#check-status).
3. If you cloned the host from another host that runs Axolet, or the root CA certificate of AxoConsole changed, reinstall Axolet. For details, see [Reinstall Axolet](https://axoflow.com/docs/axoflow/provisioning/axolet/reinstall-axolet/index.md).

## Installation problems

### AxoConsole on premises

- If you get the `[ERROR] Timed out waiting for k3s node to be Ready` error message, increase the timeout limit by running `K3S_TIMEOUT=180`, then rerun the installation script.

In case of other errors, run the following command to create a debug package from the cluster, and [contact our support team](https://axoflow.com/contact?contact_form_subject=support_request).

```shell
sudo k3s kubectl cluster-info dump --all-namespaces  --output-directory=./cluster-dump && tar -czvf cluster-dump.tar.gz cluster-dump
```

### Axoflow agent for Windows

If you run into problems when installing Axoflow agent for Windows, complete the following steps.

1. Double-check that you’re running the installation command:

   - as administrator,
   - in PowerShell, not in Cmd.
2. Set the following environment variable to store the installation logs. Adjust the directory as needed.

   ```shell
   $env:TEMP_DIR = "C:\Temp\axoflow-agent-install"
   ```
3. Re-run the installation command.
4. Open the Windows Event Viewer and check the **Windows Logs > System** events at the time of installation for errors or issues logged by the Service Control Manager.
5. Check that the installation logs of `axolet` and `axoflow-otel-collector` are available, for example, the `C:\Temp\axoflow-agent-install\axolet-windows-amd64.msi-install.log` and `C:\Temp\axoflow-agent-install\axoflow-otel-collector_<version-number>-axoflow.1_windows_x64.msi-install.log` files exist.
6. If you suspect that the installer is incorrectly reporting connectivity issues, disable the preflight check and re-run the installation command.

   ```shell
   $env:SKIP_PREFLIGHT_CHECK = 'true'
   ```
7. If needed, [contact our support team](https://axoflow.com/contact?contact_form_subject=support_request).

## Contact support

If you can’t solve the problem, [contact the Axoflow support team](https://axoflow.com/contact?contact_form_subject=support_request).

---

[A service fills the disk with logs](https://axoflow.com/docs/axoflow/troubleshoot/log-flood/index.md)

Find the service that floods the logs, free disk space, rate-limit the service in the systemd journal, and cap the journal and syslog file size.