Menu
Active1 month ago
I am new to kafka. We want to monitor and manage kafka topics. We tried different open source monitoring tools like
Both tools are good. But we are unable to make a decision which should be included in our deployment stack. Which one is better and why, and in which scenario?
'kafka manager' from yahoo looks the older one and 'kafka monitor' from LinkedIn is newer one
Products In addition to the SQL Query Tool, our company offers the following products. Kafka Tool is a GUI application for using and managing Apache Kafka clusters. It provides an intuitive UI allows you to quickly view the structure of an Apache Kafka cluster -- including the brokers, topics and consumers. To download the Kafka UI Tool for your operating system, use the links below. All versions of Kafka Tool come with a bundled JRE with the exception of the Linux version. For Linux, you must have Java 8 installed on your operating system before using Kafka Tool.
Apache Kafka Series - Learn Apache Kafka for Beginners v2 4.7 (5,054 ratings) Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately. Apache Kafka docker image for developers; with Landoop Lenses (landoop/kafka-lenses-dev) or Landoop's open source UI tools (landoop/fast-data-dev). Have a full fledged Kafka installation up and running in seconds and top it off with a modern streaming platform (only for kafka-lenses-dev), intuitive UIs and extra goodies. The kafka-topics-ui is a user interface that interacts with the kafka rest-proxy to allow browsing data from Kafka Topics. At the same time it brings visibility by providing a single entry point to explore i) Kafka data, ii) Kafka Schemas, iii) Kafka connectors and a lot more, such as partitions per topic, replication factor per topic and topic.
Kafka Monitor-
cricket_00790.8k1111 gold badges5757 silver badges136136 bronze badges
Mr.Pramod AnaraseMr.Pramod Anarase![Kafka Ui Tool For Mac Kafka Ui Tool For Mac](/uploads/1/3/3/8/133819992/141311224.jpg)
64922 gold badges88 silver badges1919 bronze badges
closed as primarily opinion-based by ChrisF♦Jul 2 at 11:58
Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.
5 Answers
Lenses
Lenses (ex Landoop) enhances Kafka with User Interface, streaming SQL engine and cluster monitoring. It enables faster monitoring of Kafka data pipelines.
Lenses (ex Landoop) enhances Kafka with User Interface, streaming SQL engine and cluster monitoring. It enables faster monitoring of Kafka data pipelines.
They provide a free all-in-one docker (Lenses Box) which can serve a single broker for up to 25M messages. Note that this is recommended for development environments.
Confluent
Another option is Confluent Enterprise which is a Kafka distribution for production environments. It also includes Control Centre, which is a management system for Apache Kafka that enables cluster monitoring and management from a User Interface.
Another option is Confluent Enterprise which is a Kafka distribution for production environments. It also includes Control Centre, which is a management system for Apache Kafka that enables cluster monitoring and management from a User Interface.
Yahoo Kafka Manager
Kafka Manager is a tool for monitoring Kafka offering less functionality compared to the aforementioned tools.
Kafka Manager is a tool for monitoring Kafka offering less functionality compared to the aforementioned tools.
KafDrop
KafDrop is a UI for monitoring Apache Kafka clusters. The tool displays information such as brokers, topics, partitions, and even lets you view messages. It is a lightweight application that runs on Spring Boot and requires very little configuration.
KafDrop is a UI for monitoring Apache Kafka clusters. The tool displays information such as brokers, topics, partitions, and even lets you view messages. It is a lightweight application that runs on Spring Boot and requires very little configuration.
LinkedIn Burrow
Burrow is a monitoring companion for Apache Kafka that provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand. An HTTP endpoint is provided to request status on demand, as well as provide other Kafka cluster information. There are also configurable notifiers that can send status out via email or HTTP calls to another service.
Burrow is a monitoring companion for Apache Kafka that provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand. An HTTP endpoint is provided to request status on demand, as well as provide other Kafka cluster information. There are also configurable notifiers that can send status out via email or HTTP calls to another service.
Kafka Tool
Kafka Tool is a GUI application for managing and using Apache Kafka clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster. It contains features geared towards both developers and administrators.
Kafka Tool is a GUI application for managing and using Apache Kafka clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster. It contains features geared towards both developers and administrators.
If you cannot afford licenses, then go for Yahoo Kafka Manager, LinkedIn Burrow or KafDrop. Confluent's and Landoop's products are the best out there, but unfortunately, they require licensing.
For more details, you can refer to my blog post Overview of UI Monitoring tools for Apache Kafka Clusters.
Giorgos MyrianthousGiorgos Myrianthous8,71044 gold badges2929 silver badges6161 bronze badges
If you want to pay for licensing and Kafka cluster support, then you can use Confluent Control Center
Alternatively, the free route would be to use JMX exporters from Datadog and/or Prometheus/Influxdb (with Grafana dashboards) to see overall system health checks (CPU, network, memory, etc)... Much more information than what you get only by monitoring Kafka processes with Kafka tools
cricket_007cricket_00790.8k1111 gold badges5757 silver badges136136 bronze badges
At my company, we used the Yahoo product, we investigated the LinkedIn product, and several others mentioned. My company ultimately chose to use Prometheus+Grafana. Everyone loves it and I'd highly recommend it.
There are two big advantages to Prometheus+Grafana. The first is it does full featured Kafka metrics ingestion+visualization+alerting but it's not limited to Kafka. While our initial needs were just to monitor Kafka, we also wanted metrics on HTTP servers+traffic, server utilization (cpu/ram/disk), and custom application level metrics. Prometheus handles all of the above. Secondly, Prometheus + Grafana are very high quality, well designed, and easy to use. A lot of other products in this space are old and complicated to work with. Prometheus + Grafana are both excellent to work with, they are very customizable, polished, and easy to use. Grafana has a very flashy + functional JavaScript interface that lets you make exactly the customized dashboards that you want. Prometheus has a very polished metric collection engine, storage engine, query language, and alerting system. Something like Yahoo Kafka Manager has much more limited functionality in all of these categories.
If you want to try Prometheus, you need to do two things:
1) install+configure the JMX->Prometheus exporter on your Kafka brokers:https://github.com/prometheus/jmx_exporter
2) Setup a Prometheus server to collect metrics + and setup a Grafana dashboard to display the graphs that you want.
I'd also say that this is just for monitoring+dashboards+alerting. For management functions, you still need other tools.
clayclay![Kafka topics ui Kafka topics ui](https://homeadvisor.tech/wp-content/uploads/2016/03/kafdrop-consumer.png)
5,77188 gold badges4747 silver badges102102 bronze badges
The Kafka ecosystem, whilst excellent in many ways does suffer from a very apparent lack of tooling. Monitoring is one such gap, but it's also a fairly broad topic.
Kafka Manager and Monitor give you insights into the health of the brokers and some metrics (throughput, latency, etc.), but you also may want to monitor consumer health and topic contents (indispensable for debugging applications or post-mortem) — neither Manager or Monitor will do that. I'm going to suggest adding some more tools to your arsenal:
- Burrow - monitors consumer group lag and raises alerts if the consumers fall behind.
- Kafdrop 3 - provides broker information and lets you browse topic contents (both text and Avro formats). This is a modern fork of the original Kafdrop project.
- Prometheus and Grafana - there's some overlap with Manager/Monitor, but Prometheus/Grafana are much more refined, have substantial community support and are easier to integrate with other tools, automatically raise alerts, etc.
Having run Kafka now for some time as a main messaging/event backbone for large microservices deployments, can't stress enough the importance of good monitoring and post-mortem tools.
Emil KoutanovEmil Koutanov
At work we use Kafdrop 3 for viewing broker information, browsing topics and consumer lag. The primary reason for Kafdrop 3 is because we lock our Kafka down in production and all producers and consumers are in a Kubernetes cluster. We don't have access to Kafka on our dev machines (the ports are not exposed to the outside world). So we run Kafdrop inside the K8s cluster and proxy to it with
kubectl proxy
.The consumer lag feature isn't as sophisticated at the that Burrow provides, but is probably good enough, especially considering that it's in a single tool.
For development (running Kafka on local), we could be using Kafka Tool, but strictly speaking it's only for non-commercial use. And besides, it's not much use if you're connecting to a production Kafka that's been locked down.
Kacey BuiKacey Bui