Kafka, mail # user - Broker does not publish offset in zookeeper - 2013-08-22, 14:42
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Broker does not publish offset in zookeeper
Hello

I am trying to monitor our kafka cluster. For that I use a zookeeper client
that reads the information of all the consumer and a jmx agent that
connects to the brokers. I try to look for discrepancies between the data
being produced and the data being consumed.

I dislike the idea of using two different connect types (jmx and zookeeper)
for this so I wonder why broker does not publish the offset into zookeeper?
I don't want the last offset, just and update every X seconds or minutes.
Is there any reason for not doing that?

Other option is to use only jmx connections but then discovering the
consumers is impossible.

Thanks

Pablo

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB