AxoSyslog 4.9.0 introduces support for ClickHouse as a destination, allowing you to send your collected data directly to a self-hosted ClickHouse database. Our community has long requested this feature, and we’re excited to deliver it in our latest release. Integrating AxoSyslog with ClickHouse opens new possibilities for efficient log data ingestion, storage, and analysis.

What is AxoSyslog?

AxoSyslog is a high-performance, open-source event (log, metrics, traces) processing solution. As a modern fork of syslog-ng, it excels at collecting, filtering, parsing, and transforming log data from various sources. Designed for cloud-native environments, AxoSyslog offers features such as:

Container Support: Easy deployment in containerized environments.
Helm Charts: Simplified Kubernetes deployments.
eBPF Integration: Efficient kernel-level data collection for minimal overhead.

AxoSyslog enables organizations to process large volumes of log data with high throughput and low latency, making it ideal for modern IT infrastructures.

What is ClickHouse?

ClickHouse is a high-performance, open-source columnar database management system optimized for real-time analytical processing. It’s capable of handling large volumes of data at incredible speeds, thanks to features like:

Columnar Storage: Efficient data compression and faster query execution.
Distributed Architecture: Scalability across multiple nodes.
Powerful SQL Support: Advanced querying capabilities for complex data analysis.

ClickHouse is widely used for analytics applications, including log analysis, due to its ability to process structured and semi-structured data efficiently.

Why is this integration important?

The integration of AxoSyslog with ClickHouse brings together two powerful technologies that complement each other perfectly:

High-Performance Data Ingestion: AxoSyslog’s efficient log processing ensures rapid ingestion of log data, which can now be seamlessly directed into ClickHouse.
Real-Time Analytics: With ClickHouse’s optimized query engine and columnar storage, users can perform real-time analytics on log data, enabling proactive monitoring and troubleshooting.
Scalability and Flexibility: Both tools are designed to handle large-scale data workloads, making them suitable for enterprises requiring scalable solutions.
Community-Driven Development: As open-source projects, AxoSyslog and ClickHouse benefit from community contributions, fostering innovation and reducing licensing costs.

This integration empowers organizations to build robust, scalable, and cost-effective log management and analytics platforms.

How to get started

To help you leverage this new integration, we’ve prepared a step-by-step guide. This example demonstrates how to:

Set up a ClickHouse server.
Configure it to accept data via gRPC.
configure AxoSyslog to send log data to ClickHouse.

Step 1: Create a Docker Network

First, create a Docker network to allow the ClickHouse server and AxoSyslog to communicate:

# Create Docker demo network
docker network create clickhouse-net

Step 2: Configure ClickHouse for gRPC Input

Create a grpc.xml configuration file for ClickHouse to enable gRPC input, and place this file in your current working directory.

<clickhouse>
  <listen_host>0.0.0.0</listen_host>
  <grpc_port>9100</grpc_port>
  <grpc>
    <enable_ssl>false</enable_ssl>
    <compression>deflate</compression>
    <compression_level>medium</compression_level>
    <max_send_message_size>-1</max_send_message_size>
    <max_receive_message_size>-1</max_receive_message_size>
    <verbose_logs>false</verbose_logs>
  </grpc>
</clickhouse>

Step 3: Run the ClickHouse Server

Start the ClickHouse server with the gRPC configuration:

# Run ClickHouse server
docker run -d \
  -p 9000:9000 \
  -p 8123:8123 \
  -p 9100:9100 \
  -v $(PWD):/etc/clickhouse-server/config.d/ \
  --network clickhouse-net \
  --name clickhouse-server \
  clickhouse/clickhouse-server

Step 4: Create the ClickHouse Destination Table

Use the ClickHouse client to create a table where AxoSyslog will send data:

# Connect to ClickHouse client
docker run -it --rm \
  --network clickhouse-net \
  clickhouse/clickhouse-client \
  --host clickhouse-server

Once connected, create the table:

CREATE TABLE my_first_table
(
  user_id UInt32,
  message String,
  timestamp DateTime,
  metric Float32
)
ENGINE = MergeTree
PRIMARY KEY (user_id, timestamp);

Exit the ClickHouse client after the table is created.

Step 5: Configure AxoSyslog

Create a syslog-ng.conf file to configure AxoSyslog:

log {
  source { example-msg-generator(freq(0.0001)); };
  destination {
    clickhouse(
      database("default")
      table("my_first_table")
      user("default")
      password("")
      url("clickhouse-server:9100")
      schema(
        "user_id" UInt32 => $R_MSEC,
        "message" String => "$MSG",
        "timestamp" DateTime => "$R_UNIXTIME",
        "metric" Float32 => 3.14
      )
      workers(4)
      batch-lines(1)
      batch-timeout(1000)
    );
  };
};

Note: Ensure the url parameter points to clickhouse-server:9100 as both containers are on the same Docker network.

Step 6: Run AxoSyslog

Start the AxoSyslog container with the custom configuration:

# Run AxoSyslog instance
docker run --rm -it \
  --network clickhouse-net \
  --entrypoint=sh \
  -v $(PWD)/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf \
  ghcr.io/axoflow/axosyslog:latest

Inside the container, start AxoSyslog:

# Start AxoSyslog
syslog-ng

AxoSyslog will now start generating example messages and send them to ClickHouse.

Step 7: Verify Data Ingestion

Use the ClickHouse client to query the ingested data:

# Connect to ClickHouse client
docker run -it --rm \
  --network clickhouse-net \
  clickhouse/clickhouse-client \
  --host clickhouse-server

Run the following SQL query:

SELECT * FROM my_first_table;

This confirms that AxoSyslog is successfully sending data to ClickHouse.

Conclusion

The addition of ClickHouse destination support in AxoSyslog 4.9.0 marks a significant milestone in our commitment to providing robust and scalable log management solutions. This integration allows users to leverage the high-performance log processing of AxoSyslog with the powerful analytical capabilities of ClickHouse.

We encourage the community to explore this new feature and share feedback. Together, we can continue to enhance and innovate within the open-source ecosystem.