Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Flume custom decorator for Rolling FileSink output bucketing


+
Dibyajyoti Ghosh 2013-03-15, 19:16
+
Mike Percy 2013-03-15, 19:38
Copy link to this message
-
Re: Flume custom decorator for Rolling FileSink output bucketing
Thanks Mike for the suggestion. The reason I am thinking of usual file
system for log storage is to avoid latency issues for file retrieval as
well as to allow users to scrape log files using grep / awk and multitude
of other powerful commands available in conventional storage.

I am now thinking of coming up with my own decorator classes for
RollingFile sink. Any pointers on how I can get started on writing my
custom decorators?

Another quick question: Can you, Mike or somebody from flume community tell
me how to use the commands documented here at:
http://archive.cloudera.com/cdh/3/flume/UserGuide/#_introducing_sink_decorators
Is this available for flume-ng distributed with Cloudera solution i.e.
flume 1.3.0?

Best and thanks a lot again,
- dib
On Fri, Mar 15, 2013 at 12:38 PM, Mike Percy <[EMAIL PROTECTED]> wrote:

> Dib, you could use the HDFS sink with a file:// URL as an option.
>
> Regards,
> Mike
>
>
>
> On Fri, Mar 15, 2013 at 12:16 PM, Dibyajyoti Ghosh <
> [EMAIL PROTECTED]> wrote:
>
> > Dear flume team,
> >
> > I am using flume 1.3.0 bundled with Cloudera 4.2.0 distribution for log
> to
> > local file system. But current implementation of FileSink doesn't have
> > inline decorators like in HDFS Sink where output can be stored to
> > directories based on event meta data e.g. hostname of the event or
> > timestamp or some other attribute in the message object.
> >
> > How can I do the same for FileSink?
> >
> >
> > Thanks a lot,
> > - dib
> >
>
+
Juhani Connolly 2013-03-18, 01:55
+
Dibyajyoti Ghosh 2013-03-18, 17:28
+
Juhani Connolly 2013-03-19, 03:15
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB