Hi folks,

I'm contemplating moving our current Flume 0.94 pipeline to Flume NG.  At
the moment we make heavy use of the "multitail" source with some custom
plugins to open up files/archives as they arrive in a specified directory,
do some parsing and then emit the parsed events.

Ideally I would replace this stage with some Flume NG counterpart.  Is
there any Flume NG source that is designed to emit events from files, or is
the suggestion that we build standalone parsers / tailers and
transmit/receive the events over some standard RPC (using Avro, for
instance)?

Best,
Jon

Hi Jon,

         Flume-NG provides an exec source that executes a given
command and consumes the output. It should serve the purpose.

Regards,
    Mohammad Tariq
On Tue, Jul 17, 2012 at 10:52 PM, Jon Bender <[EMAIL PROTECTED]> wrote:
> Hi folks,
>
> I'm contemplating moving our current Flume 0.94 pipeline to Flume NG.  At
> the moment we make heavy use of the "multitail" source with some custom
> plugins to open up files/archives as they arrive in a specified directory,
> do some parsing and then emit the parsed events.
>
> Ideally I would replace this stage with some Flume NG counterpart.  Is there
> any Flume NG source that is designed to emit events from files, or is the
> suggestion that we build standalone parsers / tailers and transmit/receive
> the events over some standard RPC (using Avro, for instance)?
>
> Best,
> Jon

Using tail command will not be useful if something goes wrong since it does
not maintain the state..
Better to implemet your own parser or application that reads the file and
sends the event to the flume over RPC client.

Regards,
Som

On Wed, Jul 18, 2012 at 1:17 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hi Jon,
>
>          Flume-NG provides an exec source that executes a given
> command and consumes the output. It should serve the purpose.
>
> Regards,
>     Mohammad Tariq
>
>
> On Tue, Jul 17, 2012 at 10:52 PM, Jon Bender <[EMAIL PROTECTED]>
> wrote:
> > Hi folks,
> >
> > I'm contemplating moving our current Flume 0.94 pipeline to Flume NG.  At
> > the moment we make heavy use of the "multitail" source with some custom
> > plugins to open up files/archives as they arrive in a specified
> directory,
> > do some parsing and then emit the parsed events.
> >
> > Ideally I would replace this stage with some Flume NG counterpart.  Is
> there
> > any Flume NG source that is designed to emit events from files, or is the
> > suggestion that we build standalone parsers / tailers and
> transmit/receive
> > the events over some standard RPC (using Avro, for instance)?
> >
> > Best,
> > Jon
>