Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Flume NG - preferred alternative to TailSource


Copy link to this message
-
Re: Flume NG - preferred alternative to TailSource
shekhar sharma 2012-07-18, 06:47
Using tail command will not be useful if something goes wrong since it does
not maintain the state..
Better to implemet your own parser or application that reads the file and
sends the event to the flume over RPC client.

Regards,
Som

On Wed, Jul 18, 2012 at 1:17 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hi Jon,
>
>          Flume-NG provides an exec source that executes a given
> command and consumes the output. It should serve the purpose.
>
> Regards,
>     Mohammad Tariq
>
>
> On Tue, Jul 17, 2012 at 10:52 PM, Jon Bender <[EMAIL PROTECTED]>
> wrote:
> > Hi folks,
> >
> > I'm contemplating moving our current Flume 0.94 pipeline to Flume NG.  At
> > the moment we make heavy use of the "multitail" source with some custom
> > plugins to open up files/archives as they arrive in a specified
> directory,
> > do some parsing and then emit the parsed events.
> >
> > Ideally I would replace this stage with some Flume NG counterpart.  Is
> there
> > any Flume NG source that is designed to emit events from files, or is the
> > suggestion that we build standalone parsers / tailers and
> transmit/receive
> > the events over some standard RPC (using Avro, for instance)?
> >
> > Best,
> > Jon
>