Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX


+
Monika Garg 2013-10-21, 12:43
+
Neha Narkhede 2013-10-21, 15:05
+
Monika Garg 2013-10-21, 16:43
+
Jun Rao 2013-10-22, 03:31
+
Monika Garg 2013-10-22, 07:55
+
Jun Rao 2013-10-22, 15:21
+
Monika Garg 2013-10-22, 17:31
+
Monika Garg 2013-10-22, 17:41
+
Joel Koshy 2013-10-22, 17:58
Copy link to this message
-
Re: Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX
Thanks a lot Joel.

On Tue, Oct 22, 2013 at 11:27 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> >
> (1)"kafka.network":type="RequestChannel",name="Processor-0-ResponseQueueSize",
> >
> "kafka.network":type="RequestChannel",name="Processor-1-ResponseQueueSize",
> >
> "kafka.network":type="RequestChannel",name="Processor-2-ResponseQueueSize"
> > Im not getting what  these Processor in (1) are actually for?
>
> Kafka's request handling utilizes a number of processor "threads" and
> responses are added to the above queue before sending to the client.
> This diagram may help:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals
>
>
> >
> (2)"kafka.server":type="FetcherLagMetrics",name="ReplicaFetcherThread-0-1-host_moni-192-168-145-119-port_9092-demo-0-ConsumerLag"
>
> This reports how far behind that replica is. The format is
>
> ReplicaFetcherThread-<fetcherid>-<leader-broker-id>-<leaderbrokerinfo>-<topic>-<partition>-ConsumerLag
>
> (Fetcher id is typically zero (since the default number of replica
> fetchers is one).)
>
> >
> >
> (3)"kafka.server":type="FetcherStats",name="ReplicaFetcherThread-0-1-host_moni-192-168-145-119-port_9092-RequestsPerSec"
> >
> (4)"kafka.server":type="FetcherStats",name="ReplicaFetcherThread-0-2-host_moni-192-168-145-176-port_9092-BytesPerSec"
> > The Mbeans (2),(3) and (4) are getting generated on my 3rd broker -
> > host_moni-192.168.145.67,kafka running on same port-9092.Any reason for
> > these.
>
> Requests per sec/byte rate for replica fetchers, but for all fetches
> to the indicated leader brokers (1 and 2).
>
>
> > I am getting some more Mbeans which are not mentioned here-
> > http://kafka.apache.org/documentation.html#monitoring
> >
> Kafka exposes a *lot* of mbeans - not all of them are listed on that
> page that you are referring to. I think the goal of the table was to
> provide a minimal set of critical metrics that people may want to set
> up monitoring/alerting for.
>
> Thanks,
>
> Joel
>
> >
> > On Tue, Oct 22, 2013 at 11:01 PM, Monika Garg <[EMAIL PROTECTED]>
> wrote:
> >
> >> Thanks Jun.
> >>
> >>
> >>
> >>
> >> On Tue, Oct 22, 2013 at 8:51 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >>
> >>> When a broker goes down, we have an optimization in the controller that
> >>> proactively shrinks ISR. The ISR shrink rate only measures the rate
> that
> >>> the leader shrinks ISR and doesn't reflect the shrinking by the
> >>> controller.
> >>> In general, having ISR shrink rate stay at 0 is a good thing since it
> >>> means
> >>> that the followers can always keep up.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Tue, Oct 22, 2013 at 12:55 AM, Monika Garg <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>> > Thanks a lot Jun.
> >>> >
> >>> > Now im getting the values in ISR expansion rate,whenever broker goes
> >>> down
> >>> > and then up.
> >>> >
> >>> > But still ISR shrink rate is always having value 0.
> >>> >
> >>> > what I have done is:
> >>> >
> >>> > I made 3 node cluster.Than created 5 partitions with replica=3.
> >>> > partitions got distributed on 1,2,3 brokers with isr= 1,2,3, and
> >>> replica =
> >>> > 1,2,3
> >>> > broker -3 is leader for 2 partitions.
> >>> >
> >>> > then i stopped one broker(say 3),then isr = 1,2 and replica = 1,2,3.
> >>> > In this case I waited for some time to get the ISR shrink value to
> get
> >>> > updated in my jconsole,but value remained 0 for all attributes of ISR
> >>> > shrink rate.
> >>> >
> >>> > Then I started broker(say 3) that i stopped earlier,what I saw is:
> >>> >
> >>> > ISR expansion rate has the values as:
> >>> >
> >>> > count -2 (on both other brokers 1,2)
> >>> > oneminuterate - 0.012 etc.
> >>> >
> >>> > then I again stopped broker-1,still isr shrink rate not getting any
> >>> > value.Can you please check?
> >>> >
> >>> > One more thing if you can please have a look:
> >>> > How the other attributes like FifteenMinuteRate,OneMinuteRate etc are
> >>> > continuosly changing for any Mbean.
> >>> >
> >>> >
> >
*Moniii*

 
+
Monika Garg 2013-10-22, 08:01
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB