kafka: Publish messages to Apache Kafka (Java implementation)

Starting with version 3.7, AxoSyslog can directly publish log messages to the Apache Kafka message bus, where subscribers can access them.

  • Since AxoSyslog uses the official Java Kafka producer, the kafka destination has significant memory usage.

Declaration:

   @include "scl.conf"
    
    kafka(
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-kafka-libraries>")
        kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
        topic("${HOST}")
    
    );

Example: Sending log data to Apache Kafka

The following example defines a kafka destination, using only the required parameters.

   @include "scl.conf"
    
    destination d_kafka {
      kafka(
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/")
        kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
        topic("${HOST}")
      );
    };

The kafka() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of AxoSyslog. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the kafka configuration snippet on GitHub.

Last modified November 20, 2024: Broken link updates (5644de9)