Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa, mail # user - How to begin with Chukwa


Copy link to this message
-
Re: How to begin with Chukwa
Bill Graham 2011-11-05, 00:08
One difference between Chukwa and Flume is that in Chukwa you configure
each component individually to know how to handle different datatypes. This
makes it fairly straight-forward to set up, but can be painful to manage
for very large installs (so I've heard). Flume centralized the configs of
"flows" which helps when managing larger installs, at the cost of
additional complexity.

On Fri, Nov 4, 2011 at 7:02 AM, AD <[EMAIL PROTECTED]> wrote:

> Are you running the agent and collector on the same machine?  Off the cuff
> its something like
>
> 1 - set your CHUKWA_HOME env variable
> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars look
> good
> 3 - edit conf/initial_adaptors with the adaptors you need.  note here that
> the dataType
> 4 - edit conf/collectors and add the hostname/ip of the collectors
> 5 - edit conf/agents and add ip/hostname of the agents
> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
> for your dataType (name from initial_adaptors)
>
> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
>
> On collector node run bin/chukwa collector  ,  tail
> ${CHUKWA_LOGS]/collector.log
>
> On collector node run bin/chukwa Demux  ,  tail ${CHUKWA_LOGS}/Demux.log
>
>
>
> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>
>> Hello AD,
>>   Yes, I have read the wiki and right now I am trying to set up a
>> Chukwa cluster using the wiki itself. Are there any other links you
>> are aware of that can help me learn and use Chukwa in a better
>> fashion?? I am trying to use Chukwa for data in "The Wellsite
>> Information Transfer Standard Markup Language (WITSML)" format.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
>> >
>> > I think it helps to read he wiki and understand the basic concepts,
>> like agents and collectors.  What kind of data do you want to parse?  I
>> have used both Flume and Chukwa myself and can say for sure Chukwa is miles
>> easier to work with and configure.  Flume may have some more advanced
>> features, but so far i found Chukwa both easy to extend from a development
>> standpoint, and much easier to configure.
>> > Have you read the wiki pages ?
>> >
>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> Hello list,
>> >>   I am totally new to Chukwa and I have to learn and start using it as
>> soon soon as possible as a part of my new project. Could any one let me
>> know how to begin with?? It would be a great favor to me. I know Flume and
>> have used it already.Also I would like to know how different is Chukwa from
>> Flume??
>> >>   Many thanks in advance.
>> >>
>> >>   Regards,
>> >>     Mohammad Tariq
>> >
>>
>
>