Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Avro client in NG as replacement for tail source in OG?


+
Chris Neal 2012-08-13, 17:43
+
Stern, Mark 2012-08-13, 20:03
+
Patrick Wendell 2012-08-13, 23:28
+
Chris Neal 2012-08-14, 14:19
+
Chris Neal 2012-08-14, 15:00
+
Chris Neal 2012-08-14, 15:36
+
Patrick Wendell 2012-08-14, 20:15
Copy link to this message
-
Re: Avro client in NG as replacement for tail source in OG?
Ah yes.  Good point.  I'll look at that.

Also, I noticed that the Log4JAppender doesn't do anything with a Pattern
Layout if it's supplied, so I added that capability.  I'll work on getting
that submitted back.

Chris

On Tue, Aug 14, 2012 at 3:15 PM, Patrick Wendell <[EMAIL PROTECTED]> wrote:

> Just a note on reliability with this approach,
>
> I'm not sure how the log4j appender handles the case where it can't
> connect to the downstream avro source (i.e. if there is a failure at
> the flume agent tier). I just looked through the code and it looks
> like it throws a FlumeException (a RuntimeException) if an append
> fails. I'm not sure if log4j eats RunitmeExceptions in appenders or
> not. If it does, this risks data loss, and if it doesn't you'll need
> to handle this possibility of an exception in your own code.
>
> - Patrick
>
> On Tue, Aug 14, 2012 at 8:36 AM, Chris Neal <[EMAIL PROTECTED]> wrote:
> > Ahh...and the best solution yet...Use
> >
> .//flume-ng-clients/flume-ng-log4jappender/target/flume-ng-log4jappender-1.3.0-SNAPSHOT.jar
> > and use the already built appender for log4j. :)  I knew there was a
> better
> > way!
> >
> > chris
> >
> >
> > On Tue, Aug 14, 2012 at 10:00 AM, Chris Neal <[EMAIL PROTECTED]> wrote:
> >>
> >> I could get around the tail -F problem of Solaris with gtail, but that
> >> doesn't address the reliability problem.
> >>
> >> I also found this:
> >>
> http://logging.apache.org/log4j/2.x/manual/appenders.html#FlumeAvroAppender
> >> which has potential, but it's in an alpha release of log4J, which won't
> fly
> >> in my production environment.
> >>
> >> Can anyone comment on the priority/timeframe of a tail source in NG?  :)
> >>
> >> Thanks!
> >>
> >>
> >> On Tue, Aug 14, 2012 at 9:19 AM, Chris Neal <[EMAIL PROTECTED]> wrote:
> >>>
> >>> Thanks Mark and Patrick,
> >>>
> >>> I had seen the ExecSource, and it's associated caveats :)  That has two
> >>> problems for me:
> >>>
> >>> 1)  It's aforementioned unreliability should the channel have errors,
> and
> >>> 2)  All my applications run on Solaris, whose tail command does not
> have
> >>> a -F equivalent from the Linux world. :(
> >>>
> >>> I noticed that the wiki here does mention a feature for NG for
> supporting
> >>> a tail source, but no JIRA is assigned to it yet.  If that page is
> still up
> >>> to date, I'm glad to see that it's at least on the roadmap!
> >>>
> >>> If anyone can think of any other creative ways to do this, please
> share.
> >>>
> >>> Much appreciated!
> >>> Chris
> >>>
> >>>
> >>> On Mon, Aug 13, 2012 at 6:28 PM, Patrick Wendell <[EMAIL PROTECTED]>
> >>> wrote:
> >>>>
> >>>> Hey Chris,
> >>>>
> >>>> Mark has got it - for the behavior you are looking, you probably want
> >>>> a tier of flume agents running where you currently have avro clients,
> >>>> using an exec source with "tail -F" as the command.
> >>>>
> >>>> Keep in minding the red warning box in the user guide related to
> >>>> sources like this. If a flume agent restarts, it may re-send or miss
> >>>> certain log entries.
> >>>>
> >>>> - Patrick
> >>>>
> >>>> On Mon, Aug 13, 2012 at 1:03 PM, Stern, Mark <[EMAIL PROTECTED]>
> wrote:
> >>>> > Use an exec source running 'tail -F'.
> >>>> > ________________________________________
> >>>> > From: Chris Neal [[EMAIL PROTECTED]]
> >>>> > Sent: Monday, August 13, 2012 8:43 PM
> >>>> > To: [EMAIL PROTECTED]
> >>>> > Subject: Avro client in NG as replacement for tail source in OG?
> >>>> >
> >>>> > Hi all.
> >>>> >
> >>>> > I have a very typical configuration:
> >>>> >
> >>>> > Application logs to log4J file.
> >>>> > FlumeNG avro-client watches the file and sends events into FlumeNG
> >>>> > Agent Tier 1
> >>>> > FlumeNG Agent Tier 1:  AvroSource to FileChannel to AvroSink
> >>>> > FlumeNG Agent Tier 2: AvroSource to FileChannel to HDFSSink
> >>>> >
> >>>> > I was noticing that after some time, the Tier 1 Agent would
> disconnect
> >>>> > from the avro-client.  What is happening is that the avro-client
> sends
> >
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB