Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka 0.8 jmxtrans + puppet


Copy link to this message
-
Re: Kafka 0.8 jmxtrans + puppet
We will be using Kafka to transport access logs from frontend cache servers to HDFS.  But!  We aren't doing this in production yet, so I wouldn't quite yet feel comfortable saying we are powered by Kafka.  I hope to get this into production more in a few weeks*, and at that time we'll write a nice little blogpost describing the whole thing.

-Andrew

*a few weeks could really mean anything in busy nonprofit time :)
On Nov 6, 2013, at 9:37 AM, Joe Stein <[EMAIL PROTECTED]> wrote:

> Yes, thank you Andrew this is great stuff.
>
> May I ask how Wikimedia is using Kafka so I can update the powered by
> https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
>
> On Wed, Nov 6, 2013 at 9:29 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Cool, thanks for sharing this!
>>
>> -Neha
>> On Nov 6, 2013 6:16 AM, "Andrew Otto" <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> I just got jmxtrans set up with Kafka 0.8 over at Wikimedia.  We're using
>>> puppet, and the https://github.com/wikimedia/puppet-kafka module to
>>> install and set up Kafka.  This module now comes with a
>>> kafka::server::jmxtrans class which will automatically set up jmxtrans
>> with
>>> some nice Kafka queries.
>>>
>>> If you aren't using puppet, you can use the rendered example jmxtrans
>> JSON
>>> config here:
>>>
>>>
>>>
>> https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md
>>>
>>> The exact queries will probably change as we get more operational
>>> experience and think of more things we want to monitor.
>>>
>>> Hope this is useful to someone!
>>>
>>> -Andrew Otto
>>>
>>>
>>>
>>