Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Ganglia Metrics Reporter


Copy link to this message
-
Re: Ganglia Metrics Reporter
Hmm,

It looks as though adding multicast support is more an issue with the Metrics code rather than with kafka-ganglia.  The only way I could see to tell GangliaReporter to use multicast when sending messages was to pass it a GangliaMessageBuilder that constructed GangliaMessages using a proper MulticastSocket which sets the multicast TTL.

I attempted to do this like this:
https://gist.github.com/ottomata/7269312

However, GangliaMessageBuilder is not a public class, so I can't extend it as MulticastGangliaMessageBuilder.

I'm not sure what to do at this point.  I'd rather use this library but, it looks like I might have to go with jmxtrans after all.  Thoughts?

-Andrew
On Oct 29, 2013, at 2:38 PM, Maxime Brugidou <[EMAIL PROTECTED]> wrote:

> Hi Andrew, how do you plan to use metrics 3.0.1? The current Kafka 0.8
> version uses 2.2 AFAIK so this is going to require a new Kafka version.
>
> I don't really use multicast TTL muself but you are right that this should
> be configurable and I'll definitely accept pull request going in that
> direction. It's a tiny piece of code so it shouldn't be hard to dig it.
>
> Thanks for using this small plugin and let me know what you plan to do.
>
> Cheers
> On Oct 29, 2013 2:00 PM, "Andrew Otto" <[EMAIL PROTECTED]> wrote:
>
>> Hi Maxime,
>>
>> I'm using this at the Wikimedia Foundation to send Kafka Broker metrics to
>> Ganglia.  However, we use Ganglia in multicast mode.  This mostly seems to
>> work with your code, but the ttl on the multicast packets gets set at 1.
>> We have sometimes have multiple levels of ganglia aggregators, and with
>> ttl=1, the multicast packets don't make it to the proper aggregators.
>>
>> I'm looking into either forking or rewriting this library using Codahale
>> Metrics v 3.0.1, and supporting multicast more explicitly.  Is this
>> something you could do better/faster than me, or should I proceed? :)
>>
>> -Andrew Otto
>>
>> (Thanks for writing this, btw!)
>>
>>
>> On Aug 22, 2013, at 11:42 AM, Maxime Brugidou <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Hi all,
>>>
>>> Since I couldn't find any other way to publish kafka metrics to ganglia
>>> from kafka 0.8 (beta), I just published on github a super-simple ganglia
>>> metrics reporter for Kafka. It is configurable through the kafka config
>>> file and you can use it on the broker side and on your
>> consumers/producers.
>>> There is also a feature to exclude some metrics with a regex (useful if
>> you
>>> have many topics/partions).
>>>
>>> Here it is on github: https://github.com/criteo/kafka-ganglia
>>>
>>> Let me know if you have issues/questions. This is using metrics-ganglia
>>> 2.2.0 directly so it is not a ganglia plugin but rather a kafka "add-on".
>>>
>>> I don't know the proper way to distribute this yet for installation on
>>> brokers, it could also be part of some contrib kafka code.
>>>
>>> Cheers,
>>> Maxime
>>
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB