Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Ganglia Metrics Reporter


Copy link to this message
-
Re: Ganglia Metrics Reporter
Hmm,

It looks as though adding multicast support is more an issue with the Metrics code rather than with kafka-ganglia.  The only way I could see to tell GangliaReporter to use multicast when sending messages was to pass it a GangliaMessageBuilder that constructed GangliaMessages using a proper MulticastSocket which sets the multicast TTL.

I attempted to do this like this:
https://gist.github.com/ottomata/7269312

However, GangliaMessageBuilder is not a public class, so I can't extend it as MulticastGangliaMessageBuilder.

I'm not sure what to do at this point.  I'd rather use this library but, it looks like I might have to go with jmxtrans after all.  Thoughts?

-Andrew
On Oct 29, 2013, at 2:38 PM, Maxime Brugidou <[EMAIL PROTECTED]> wrote:

> Hi Andrew, how do you plan to use metrics 3.0.1? The current Kafka 0.8
> version uses 2.2 AFAIK so this is going to require a new Kafka version.
>
> I don't really use multicast TTL muself but you are right that this should
> be configurable and I'll definitely accept pull request going in that
> direction. It's a tiny piece of code so it shouldn't be hard to dig it.
>
> Thanks for using this small plugin and let me know what you plan to do.
>
> Cheers
> On Oct 29, 2013 2:00 PM, "Andrew Otto" <[EMAIL PROTECTED]> wrote:
>
>> Hi Maxime,
>>
>> I'm using this at the Wikimedia Foundation to send Kafka Broker metrics to
>> Ganglia.  However, we use Ganglia in multicast mode.  This mostly seems to
>> work with your code, but the ttl on the multicast packets gets set at 1.
>> We have sometimes have multiple levels of ganglia aggregators, and with
>> ttl=1, the multicast packets don't make it to the proper aggregators.
>>
>> I'm looking into either forking or rewriting this library using Codahale
>> Metrics v 3.0.1, and supporting multicast more explicitly.  Is this
>> something you could do better/faster than me, or should I proceed? :)
>>
>> -Andrew Otto
>>
>> (Thanks for writing this, btw!)
>>
>>
>> On Aug 22, 2013, at 11:42 AM, Maxime Brugidou <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Hi all,
>>>
>>> Since I couldn't find any other way to publish kafka metrics to ganglia
>>> from kafka 0.8 (beta), I just published on github a super-simple ganglia
>>> metrics reporter for Kafka. It is configurable through the kafka config
>>> file and you can use it on the broker side and on your
>> consumers/producers.
>>> There is also a feature to exclude some metrics with a regex (useful if
>> you
>>> have many topics/partions).
>>>
>>> Here it is on github: https://github.com/criteo/kafka-ganglia
>>>
>>> Let me know if you have issues/questions. This is using metrics-ganglia
>>> 2.2.0 directly so it is not a ganglia plugin but rather a kafka "add-on".
>>>
>>> I don't know the proper way to distribute this yet for installation on
>>> brokers, it could also be part of some contrib kafka code.
>>>
>>> Cheers,
>>> Maxime
>>