Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Broker does not publish offset in zookeeper


+
Pablo Barrera González 2013-08-22, 14:42
Copy link to this message
-
Re: Broker does not publish offset in zookeeper
Are you trying to monitor the lag in the consumer? If so, you can use the
maxLag/minFetch JMX described in
http://kafka.apache.org/documentation.html#monitoring

Thanks,

Jun
On Thu, Aug 22, 2013 at 7:41 AM, Pablo Barrera González <
[EMAIL PROTECTED]> wrote:

> Hello
>
> I am trying to monitor our kafka cluster. For that I use a zookeeper client
> that reads the information of all the consumer and a jmx agent that
> connects to the brokers. I try to look for discrepancies between the data
> being produced and the data being consumed.
>
> I dislike the idea of using two different connect types (jmx and zookeeper)
> for this so I wonder why broker does not publish the offset into zookeeper?
> I don't want the last offset, just and update every X seconds or minutes.
> Is there any reason for not doing that?
>
> Other option is to use only jmx connections but then discovering the
> consumers is impossible.
>
> Thanks
>
> Pablo
>

 
+
Pablo Barrera González 2013-08-22, 15:42
+
Jun Rao 2013-08-23, 03:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB