Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa, mail # dev - allocate customized information via chukwa


Copy link to this message
-
Re: allocate customized information via chukwa
Eric Yang 2012-03-30, 14:51
Hi,

Chukwa is a generic collection system.  It should be possible to extend it
to collect your log files and implement a demur parser for your log files.

Regards,
Eric

On Friday, March 30, 2012, xzer wrote:

> Hi,
>
> Thanks for your reply, I read about your advised projects, and I
> believe scribe or flume is for my case.
>
> But I found that In flume, there is no a defined tier for post
> processing just as demux jobs in chukwa, I know I can do them by an
> individual module of my system but I believe a built-in post
> processing conception is better.
>
> Another way, In scribe, it seems that there are things called
> continuous copier and parallel tailer for post processing but it is
> not clear that how they work.
>
> My system is not big enough as facebook or google, it has small or
> middle level load and there are only several servers, I need a
> replicatable, speedy and simple solution for log allocating. I found
> the architecture of chukwa is clear and simple for a middle  scale
> system despite of what it is really doing.
>
> I think scribe is too complex for me, and I am interested in whether
> it is possible to reuse the control follow of chukwa disregard where I
> save my data to. I would do more work on those frameworks but I think
> to abstract the control follow of chukwa is a good idea because it is
> simple enough for a small system.
>
> Best Regards
>
> xzer
>
> 在 2012年3月30日 下午2:29��Jiaqi Tan <[EMAIL PROTECTED] <javascript:;>> 写道��
> > Hi,
> >
> > On Fri, Mar 30, 2012 at 1:04 PM, xzer <[EMAIL PROTECTED] <javascript:;>>
> wrote:
> >> Hi,
> >>
> >> I am working for a solution for allocating statistics information from
> >> our own systems, I found that the architecture of chuka is a good one
> >> for my work. I have 2 questions here:
> >>
> >> 1. I found that chuka is designed for allocate information of hadoop
> >> clusters not for a common useage of information allocating, is that
> >> right? or I missed something?
> >
> > Yes and no. You can run Chukwa collectors anywhere, but there are
> > quite a number of plugins designed for collecting Hadoop data. Also,
> > Chukwa deals with large volumes of collected data by writing to HDFS
> > (or HBase?), so it uses Hadoop as well.
> >
> >>
> >> 2. If I did not miss anything, is it possible that reforming chuka to
> >> a common framework for allocating information?
> >
> > There are a number of other frameworks for collecting log data and
> > other system information such as Facebook's Scribe, Cloudera's Flume,
> > and even more traditional systems such as Nagios or Ganglia.
> >
> >>
> >> Best Regards
> >>
> >> xzer
> >
> > Hope this helps,
> > Jiaqi
>