Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # dev >> allocate customized information via chukwa


Copy link to this message
-
Re: allocate customized information via chukwa
Hi,

Chukwa is a generic collection system.  It should be possible to extend it
to collect your log files and implement a demur parser for your log files.

Regards,
Eric

On Friday, March 30, 2012, xzer wrote:

> Hi,
>
> Thanks for your reply, I read about your advised projects, and I
> believe scribe or flume is for my case.
>
> But I found that In flume, there is no a defined tier for post
> processing just as demux jobs in chukwa, I know I can do them by an
> individual module of my system but I believe a built-in post
> processing conception is better.
>
> Another way, In scribe, it seems that there are things called
> continuous copier and parallel tailer for post processing but it is
> not clear that how they work.
>
> My system is not big enough as facebook or google, it has small or
> middle level load and there are only several servers, I need a
> replicatable, speedy and simple solution for log allocating. I found
> the architecture of chukwa is clear and simple for a middle  scale
> system despite of what it is really doing.
>
> I think scribe is too complex for me, and I am interested in whether
> it is possible to reuse the control follow of chukwa disregard where I
> save my data to. I would do more work on those frameworks but I think
> to abstract the control follow of chukwa is a good idea because it is
> simple enough for a small system.
>
> Best Regards
>
> xzer
>
> 在 2012年3月30日 下午2:29��Jiaqi Tan <[EMAIL PROTECTED] <javascript:;>> 写道��
> > Hi,
> >
> > On Fri, Mar 30, 2012 at 1:04 PM, xzer <[EMAIL PROTECTED] <javascript:;>>
> wrote:
> >> Hi,
> >>
> >> I am working for a solution for allocating statistics information from
> >> our own systems, I found that the architecture of chuka is a good one
> >> for my work. I have 2 questions here:
> >>
> >> 1. I found that chuka is designed for allocate information of hadoop
> >> clusters not for a common useage of information allocating, is that
> >> right? or I missed something?
> >
> > Yes and no. You can run Chukwa collectors anywhere, but there are
> > quite a number of plugins designed for collecting Hadoop data. Also,
> > Chukwa deals with large volumes of collected data by writing to HDFS
> > (or HBase?), so it uses Hadoop as well.
> >
> >>
> >> 2. If I did not miss anything, is it possible that reforming chuka to
> >> a common framework for allocating information?
> >
> > There are a number of other frameworks for collecting log data and
> > other system information such as Facebook's Scribe, Cloudera's Flume,
> > and even more traditional systems such as Nagios or Ganglia.
> >
> >>
> >> Best Regards
> >>
> >> xzer
> >
> > Hope this helps,
> > Jiaqi
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB