Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Multiple consumer connectors in process - lag metrics?


+
Jason Rosenberg 2013-11-15, 03:48
+
Neha Narkhede 2013-11-15, 14:59
Copy link to this message
-
Re: Multiple consumer connectors in process - lag metrics?
Hmmm.....I'm not sure I see how that will work?  If using multiple
connectors (and thus multiple ConsumerFetcherManagers?).

We were seeing the MaxLag at 0, with multiple connectors/same groupId, even
though we were quite sure there was significant lag in some of the
connectors....

Jason
On Fri, Nov 15, 2013 at 9:58 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> I think ConsumerFetcherManager metric will report data for all of the
> connectors with the same group.id transparently.
>
> Thanks,
> Neha
> On Nov 14, 2013 7:48 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > We are experimenting with having multiple consumer connectors running in
> > the same process, under the same groupId (but with different topic
> > filters).
> >
> > I'm wondering what the expected effect of this is with metrics, like
> > ConsumerFetcherManager.<groupid>-MaxLag
> >
> > It looks like in AbstractFetcherManager, that it creates a yammer Gauge
> for
> > this, probably repeatedly for each connector created?
> >
> > I'm not sure this metric will do the right thing?  Will it just show the
> > first (or the last one) registered?
> >
> > Jason
> >
>

 
+
Joel Koshy 2013-11-15, 22:29
+
Jun Rao 2013-11-16, 07:18
+
Jason Rosenberg 2013-11-16, 15:21
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB