kafka: Publish messages to Apache Kafka (Java implementation)

Starting with version 3.7, AxoSyslog can directly publish log messages to the Apache Kafka message bus, where subscribers can access them.

• Since AxoSyslog uses the official Java Kafka producer, the kafka destination has significant memory usage.

Declaration:

   @include "scl.conf"
    
    kafka(
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-kafka-libraries>")
        kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
        topic("${HOST}")
    
    );

Example: Sending log data to Apache Kafka

The following example defines a kafka destination, using only the required parameters.

   @include "scl.conf"
    
    destination d_kafka {
      kafka(
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/")
        kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
        topic("${HOST}")
      );
    };

• To install the software required for the kafka destination, see Prerequisites.

• For details on how the kafka destination works, see How AxoSyslog interacts with Apache Kafka.

• For the list of options, see Kafka destination options.

The kafka() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of AxoSyslog. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the kafka configuration snippet on GitHub. For details on extending AxoSyslog in Java, see the Getting started with syslog-ng development guide.