Setting up Kafka and ELK integration for APM Stream

To set up Kafka and ELK to display APM Stream data:

  1. Download and install/unzip Apache ZooKeeper, if necessary.

  1. Download, install/unzip, and configure Apache Kafka, if necessary.

  1. Create a Kafka topic for APM Stream.

    To create a Kafka topic named tidalapmtopic, enter this command:

    kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic tidalapmtopic

    Note: This name must match the topic name specified in the env-registry.xml file when installing or configuring APM Stream.

  1. Download and install or unzip the Elastic Stack, which includes ELK (Elasticsearch, Logstash, and Kibana).

  1. Configure logstash.conf to use the Kafka message as input and Elasticsearch as output, similar to this:

    input { kafka {
    codec => "plain"
    bootstrap_servers => "localhost:9092" topics => ["tidalapmtopic"]
    tags => ["tidalapmtopic"]
    }
    }
    output { elasticsearch {
    hosts => ["localhost:9200"] index => "tidalapmindex"
    }
    }

  1. Add Visualization in Kibana using the topic name (tidalapmindex, in this case) as the filter.