Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Avro client in NG as replacement for tail source in OG?


Copy link to this message
-
Re: Avro client in NG as replacement for tail source in OG?
Chris Neal 2012-08-15, 14:12
Ah yes.  Good point.  I'll look at that.

Also, I noticed that the Log4JAppender doesn't do anything with a Pattern
Layout if it's supplied, so I added that capability.  I'll work on getting
that submitted back.

Chris

On Tue, Aug 14, 2012 at 3:15 PM, Patrick Wendell <[EMAIL PROTECTED]> wrote:

> Just a note on reliability with this approach,
>
> I'm not sure how the log4j appender handles the case where it can't
> connect to the downstream avro source (i.e. if there is a failure at
> the flume agent tier). I just looked through the code and it looks
> like it throws a FlumeException (a RuntimeException) if an append
> fails. I'm not sure if log4j eats RunitmeExceptions in appenders or
> not. If it does, this risks data loss, and if it doesn't you'll need
> to handle this possibility of an exception in your own code.
>
> - Patrick
>
> On Tue, Aug 14, 2012 at 8:36 AM, Chris Neal <[EMAIL PROTECTED]> wrote:
> > Ahh...and the best solution yet...Use
> >
> .//flume-ng-clients/flume-ng-log4jappender/target/flume-ng-log4jappender-1.3.0-SNAPSHOT.jar
> > and use the already built appender for log4j. :)  I knew there was a
> better
> > way!
> >
> > chris
> >
> >
> > On Tue, Aug 14, 2012 at 10:00 AM, Chris Neal <[EMAIL PROTECTED]> wrote:
> >>
> >> I could get around the tail -F problem of Solaris with gtail, but that
> >> doesn't address the reliability problem.
> >>
> >> I also found this:
> >>
> http://logging.apache.org/log4j/2.x/manual/appenders.html#FlumeAvroAppender
> >> which has potential, but it's in an alpha release of log4J, which won't
> fly
> >> in my production environment.
> >>
> >> Can anyone comment on the priority/timeframe of a tail source in NG?  :)
> >>
> >> Thanks!
> >>
> >>
> >> On Tue, Aug 14, 2012 at 9:19 AM, Chris Neal <[EMAIL PROTECTED]> wrote:
> >>>
> >>> Thanks Mark and Patrick,
> >>>
> >>> I had seen the ExecSource, and it's associated caveats :)  That has two
> >>> problems for me:
> >>>
> >>> 1)  It's aforementioned unreliability should the channel have errors,
> and
> >>> 2)  All my applications run on Solaris, whose tail command does not
> have
> >>> a -F equivalent from the Linux world. :(
> >>>
> >>> I noticed that the wiki here does mention a feature for NG for
> supporting
> >>> a tail source, but no JIRA is assigned to it yet.  If that page is
> still up
> >>> to date, I'm glad to see that it's at least on the roadmap!
> >>>
> >>> If anyone can think of any other creative ways to do this, please
> share.
> >>>
> >>> Much appreciated!
> >>> Chris
> >>>
> >>>
> >>> On Mon, Aug 13, 2012 at 6:28 PM, Patrick Wendell <[EMAIL PROTECTED]>
> >>> wrote:
> >>>>
> >>>> Hey Chris,
> >>>>
> >>>> Mark has got it - for the behavior you are looking, you probably want
> >>>> a tier of flume agents running where you currently have avro clients,
> >>>> using an exec source with "tail -F" as the command.
> >>>>
> >>>> Keep in minding the red warning box in the user guide related to
> >>>> sources like this. If a flume agent restarts, it may re-send or miss
> >>>> certain log entries.
> >>>>
> >>>> - Patrick
> >>>>
> >>>> On Mon, Aug 13, 2012 at 1:03 PM, Stern, Mark <[EMAIL PROTECTED]>
> wrote:
> >>>> > Use an exec source running 'tail -F'.
> >>>> > ________________________________________
> >>>> > From: Chris Neal [[EMAIL PROTECTED]]
> >>>> > Sent: Monday, August 13, 2012 8:43 PM
> >>>> > To: [EMAIL PROTECTED]
> >>>> > Subject: Avro client in NG as replacement for tail source in OG?
> >>>> >
> >>>> > Hi all.
> >>>> >
> >>>> > I have a very typical configuration:
> >>>> >
> >>>> > Application logs to log4J file.
> >>>> > FlumeNG avro-client watches the file and sends events into FlumeNG
> >>>> > Agent Tier 1
> >>>> > FlumeNG Agent Tier 1:  AvroSource to FileChannel to AvroSink
> >>>> > FlumeNG Agent Tier 2: AvroSource to FileChannel to HDFSSink
> >>>> >
> >>>> > I was noticing that after some time, the Tier 1 Agent would
> disconnect
> >>>> > from the avro-client.  What is happening is that the avro-client
> sends
> >