Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Multiple consumer connectors in process - lag metrics?


Copy link to this message
-
Re: Multiple consumer connectors in process - lag metrics?
Jason Rosenberg 2013-11-15, 17:53
Hmmm.....I'm not sure I see how that will work?  If using multiple
connectors (and thus multiple ConsumerFetcherManagers?).

We were seeing the MaxLag at 0, with multiple connectors/same groupId, even
though we were quite sure there was significant lag in some of the
connectors....

Jason
On Fri, Nov 15, 2013 at 9:58 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> I think ConsumerFetcherManager metric will report data for all of the
> connectors with the same group.id transparently.
>
> Thanks,
> Neha
> On Nov 14, 2013 7:48 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > We are experimenting with having multiple consumer connectors running in
> > the same process, under the same groupId (but with different topic
> > filters).
> >
> > I'm wondering what the expected effect of this is with metrics, like
> > ConsumerFetcherManager.<groupid>-MaxLag
> >
> > It looks like in AbstractFetcherManager, that it creates a yammer Gauge
> for
> > this, probably repeatedly for each connector created?
> >
> > I'm not sure this metric will do the right thing?  Will it just show the
> > first (or the last one) registered?
> >
> > Jason
> >
>