Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Flume custom decorator for Rolling FileSink output bucketing


+
Dibyajyoti Ghosh 2013-03-15, 19:16
+
Mike Percy 2013-03-15, 19:38
Copy link to this message
-
Re: Flume custom decorator for Rolling FileSink output bucketing
Dibyajyoti Ghosh 2013-03-15, 21:20
Thanks Mike for the suggestion. The reason I am thinking of usual file
system for log storage is to avoid latency issues for file retrieval as
well as to allow users to scrape log files using grep / awk and multitude
of other powerful commands available in conventional storage.

I am now thinking of coming up with my own decorator classes for
RollingFile sink. Any pointers on how I can get started on writing my
custom decorators?

Another quick question: Can you, Mike or somebody from flume community tell
me how to use the commands documented here at:
http://archive.cloudera.com/cdh/3/flume/UserGuide/#_introducing_sink_decorators
Is this available for flume-ng distributed with Cloudera solution i.e.
flume 1.3.0?

Best and thanks a lot again,
- dib
On Fri, Mar 15, 2013 at 12:38 PM, Mike Percy <[EMAIL PROTECTED]> wrote:

> Dib, you could use the HDFS sink with a file:// URL as an option.
>
> Regards,
> Mike
>
>
>
> On Fri, Mar 15, 2013 at 12:16 PM, Dibyajyoti Ghosh <
> [EMAIL PROTECTED]> wrote:
>
> > Dear flume team,
> >
> > I am using flume 1.3.0 bundled with Cloudera 4.2.0 distribution for log
> to
> > local file system. But current implementation of FileSink doesn't have
> > inline decorators like in HDFS Sink where output can be stored to
> > directories based on event meta data e.g. hostname of the event or
> > timestamp or some other attribute in the message object.
> >
> > How can I do the same for FileSink?
> >
> >
> > Thanks a lot,
> > - dib
> >
>
+
Juhani Connolly 2013-03-18, 01:55
+
Dibyajyoti Ghosh 2013-03-18, 17:28
+
Juhani Connolly 2013-03-19, 03:15