Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Ganglia Metrics Reporter


Copy link to this message
-
Re: Ganglia Metrics Reporter
Maxime Brugidou 2013-10-29, 18:38
Hi Andrew, how do you plan to use metrics 3.0.1? The current Kafka 0.8
version uses 2.2 AFAIK so this is going to require a new Kafka version.

I don't really use multicast TTL muself but you are right that this should
be configurable and I'll definitely accept pull request going in that
direction. It's a tiny piece of code so it shouldn't be hard to dig it.

Thanks for using this small plugin and let me know what you plan to do.

Cheers
On Oct 29, 2013 2:00 PM, "Andrew Otto" <[EMAIL PROTECTED]> wrote:

> Hi Maxime,
>
> I'm using this at the Wikimedia Foundation to send Kafka Broker metrics to
> Ganglia.  However, we use Ganglia in multicast mode.  This mostly seems to
> work with your code, but the ttl on the multicast packets gets set at 1.
>  We have sometimes have multiple levels of ganglia aggregators, and with
> ttl=1, the multicast packets don't make it to the proper aggregators.
>
> I'm looking into either forking or rewriting this library using Codahale
> Metrics v 3.0.1, and supporting multicast more explicitly.  Is this
> something you could do better/faster than me, or should I proceed? :)
>
> -Andrew Otto
>
> (Thanks for writing this, btw!)
>
>
> On Aug 22, 2013, at 11:42 AM, Maxime Brugidou <[EMAIL PROTECTED]>
> wrote:
>
> > Hi all,
> >
> > Since I couldn't find any other way to publish kafka metrics to ganglia
> > from kafka 0.8 (beta), I just published on github a super-simple ganglia
> > metrics reporter for Kafka. It is configurable through the kafka config
> > file and you can use it on the broker side and on your
> consumers/producers.
> > There is also a feature to exclude some metrics with a regex (useful if
> you
> > have many topics/partions).
> >
> > Here it is on github: https://github.com/criteo/kafka-ganglia
> >
> > Let me know if you have issues/questions. This is using metrics-ganglia
> > 2.2.0 directly so it is not a ganglia plugin but rather a kafka "add-on".
> >
> > I don't know the proper way to distribute this yet for installation on
> > brokers, it could also be part of some contrib kafka code.
> >
> > Cheers,
> > Maxime
>