CMAK README

The project's README from github.com/yahoo/CMAK. As one markdown file: README.md.

CMAK (Cluster Manager for Apache Kafka, previously known as Kafka Manager)

CMAK (previously known as Kafka Manager) is a tool for managing Apache Kafka clusters. See below for details about the name change.

CMAK supports the following:

Cluster Management

cluster

Topic List

topic

Topic View

topic

Consumer List View

consumer

Consumed Topic View

consumer

Broker List

broker

Broker View

broker

Requirements

  1. Kafka 0.8.. or 0.9.. or 0.10.. or 0.11..
  2. Java 11+

Configuration

The minimum configuration is the zookeeper hosts which are to be used for CMAK (pka kafka manager) state. This can be found in the application.conf file in conf directory. The same file will be packaged in the distribution zip file; you may modify settings after unzipping the file on the desired server.

cmak.zkhosts="my.zookeeper.host.com:2181"

You can specify multiple zookeeper hosts by comma delimiting them, like so:

cmak.zkhosts="my.zookeeper.host.com:2181,other.zookeeper.host.com:2181"

Alternatively, use the environment variable ZK_HOSTS if you don't want to hardcode any values.

ZK_HOSTS="my.zookeeper.host.com:2181"

You can optionally enable/disable the following functionality by modifying the default list in application.conf :

application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature"]

Consider setting these parameters for larger clusters with jmx enabled :

Here is an example for a kafka cluster with 10 brokers, 100 topics, with each topic having 10 partitions giving 1000 total partitions with JMX enabled :

The follow control consumer offset cache's thread pool and queue :

You should increase the above for large # of consumers with consumer polling enabled. Though it mainly affects ZK based consumer polling.

Kafka managed consumer offset is now consumed by KafkaManagedOffsetCache from the "__consumer_offsets" topic. Note, this has not been tested with large number of offsets being tracked. There is a single thread per cluster consuming this topic so it may not be able to keep up on large # of offsets being pushed to the topic.

Authenticating a User with LDAP

Warning, you need to have SSL configured with CMAK (pka Kafka Manager) to ensure your credentials aren't passed unencrypted. Authenticating a User with LDAP is possible by passing the user credentials with the Authorization header. LDAP authentication is done on first visit, if successful, a cookie is set. On next request, the cookie value is compared with credentials from Authorization header. LDAP support is through the basic authentication filter.

  1. Configure basic authentication
  1. Encryption parameters (optional, otherwise randomly generated on startup) :
  1. Configure LDAP / LDAP + StartTLS / LDAPS authentication

Note: LDAP is unencrypted and insecure. LDAPS is a commonly implemented extension that implements an encryption layer in a manner similar to how HTTPS adds encryption to an HTTP. LDAPS has not been documented, and the specification is not formally defined anywhere. LDAP + StartTLS is the currently recommended way to start an encrypted channel, and it upgrades an existing LDAP connection to achieve this encryption.

  1. (Optional) Limit access to a specific LDAP Group

Example (Online LDAP Test Server):

Deployment

The command below will create a zip file which can be used to deploy the application.

./sbt clean dist

Please refer to play framework documentation on production deployment/configuration.

If java is not in your path, or you need to build against a specific java version, please use the following (the example assumes zulu java11):

$ PATH=/usr/lib/jvm/zulu-11-amd64/bin:$PATH \
  JAVA_HOME=/usr/lib/jvm/zulu-11-amd64 \
  /path/to/sbt -java-home /usr/lib/jvm/zulu-11-amd64 clean dist

This ensures that the 'java' and 'javac' binaries in your path are first looked up in the correct location. Next, for all downstream tools that only listen to JAVA_HOME, it points them to the java11 location. Lastly, it tells sbt to use the java11 location as well.

Starting the service

After extracting the produced zipfile, and changing the working directory to it, you can run the service like this:

$ bin/cmak

By default, it will choose port 9000. This is overridable, as is the location of the configuration file. For example:

$ bin/cmak -Dconfig.file=/path/to/application.conf -Dhttp.port=8080

Again, if java is not in your path, or you need to run against a different version of java, add the -java-home option as follows:

$ bin/cmak -java-home /usr/lib/jvm/zulu-11-amd64

Starting the service with Security

To add JAAS configuration for SASL, add the config file location at start:

$ bin/cmak -Djava.security.auth.login.config=/path/to/my-jaas.conf

NOTE: Make sure the user running CMAK (pka kafka manager) has read permissions on the jaas config file

Packaging

If you'd like to create a Debian or RPM package instead, you can run one of:

sbt debian:packageBin

sbt rpm:packageBin

Credits

Most of the utils code has been adapted to work with Apache Curator from Apache Kafka.

Name and Management

CMAK was renamed from its previous name due to this issue. CMAK is designed to be used with Apache Kafka and is offered to support the needs of the Kafka community. This project is currently managed by employees at Verizon Media and the community who supports this project.

License

Licensed under the terms of the Apache License 2.0. See accompanying LICENSE file for terms.

Consumer/Producer Lag

Producer offset is polled. Consumer offset is read from the offset topic for Kafka based consumers. This means the reported lag may be negative since we are consuming offset from the offset topic faster then polling the producer offset. This is normal and not a problem.

Migration from Kafka Manager to CMAK

  1. Copy config files from old version to new version (application.conf, consumer.properties)
  2. Change start script to use bin/cmak instead of bin/kafka-manager

CMAK (yahoo/CMAK)

Searching Database…

11.9K ·GitHub
CMAK
这是什么?CMAK(Cluster Manager for Apache Kafka,以前叫 Kafka Manager)是一个自托管的网页工具,用来管理 Apache Kafka 集群,可以在一个界面里管理多个集群。
它是部署在服务器上的网页控制台,还是要在本地安装的东西?它是一个网页服务:部署在有 Java 11 的服务器上,通过浏览器界面操作你的集群,默认监听 9000 端口(端口可以改)。
怎么把它跑起来?先用 ./sbt clean dist 打一个部署压缩包,解压后,把 conf/application.conf 里的 cmak.zkhosts(或 ZKHOSTS 环境变量)指向你的 ZooKeeper 主机,然后启动 bin/cmak。也可以构建 Debian 和 RPM 安装包。
要让它正常工作,我需要准备什么?需要 Java 11 或更高版本,以及一个 Kafka 集群(支持 0.8、0.9、0.10 或 0.11 版本),它的 ZooKeeper 必须可以访问,因为 CMAK 就是通过 ZooKeeper 读取集群状态的。
能通过它删除 topic 吗?只有在 Kafka 0.8.2 及以上版本,并且 broker 配置里设置了 delete.topic.enable=true 时才可以;topic 列表会标记出那些已排队待删除的 topic。
除了查看集群状态,它还能对我的集群做什么?能做的很多:它可以管理多个集群,查看 topic、消费者、偏移量、broker 以及副本和分区的分布情况,创建或删除 topic,增加分区,生成分区分配方案,执行分区重分配和首选副本选举,还可以选择性地通过 JMX 轮询 broker 和 topic 的监控指标。
可以要求用户先登录吗?可以——支持基于 LDAP 的基本认证,包括普通 LDAP、LDAPS,以及推荐的 LDAP + StartTLS,还可以把访问权限限制到某个特定的 LDAP 组。README 提醒必须配置 SSL,以免登录凭据以明文传输。
为什么叫 CMAK 而不叫 Kafka Manager?由于命名方面的问题,项目从 Kafka Manager 改名为 CMAK(Cluster Manager for Apache Kafka),目前由 Verizon Media 的员工和社区共同维护。
它支持使用 SASL 安全认证的 Kafka 集群吗?支持。启动服务时带上 JAAS 配置:bin/cmak -Djava.security.auth.login.config=/path/to/my-jaas.conf,注意运行 CMAK 的用户要有读取该文件的权限。
What are similar projects?kafka, kafka-ui, doris, zookeeper, redpanda, BigData-Notes, flink, hadoop
Which topics is it in?cluster-management, zookeeper, ldap, scala, apache, kafka, jvm, big-data, sysadmin, distributed-systems, streaming, dashboard
Creator
yahoo
Size
4 MB
Stars
11.9K
Updated
CommitDec 12Release3.0.0.6Apr 29
Age
11y
Commits · 90 Days