Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Avro client in NG as replacement for tail source in OG?

Copy link to this message
Avro client in NG as replacement for tail source in OG?
Chris Neal 2012-08-13, 17:43
Hi all.

I have a very typical configuration:

Application logs to log4J file.
FlumeNG avro-client watches the file and sends events into FlumeNG Agent
Tier 1
FlumeNG Agent Tier 1:  AvroSource to FileChannel to AvroSink
FlumeNG Agent Tier 2: AvroSource to FileChannel to HDFSSink

I was noticing that after some time, the Tier 1 Agent would disconnect from
the avro-client.  What is happening is that the avro-client sends events to
the Tier 1 Agent as fast as it can, and when it reaches the end of the
file, it exits.  The problem is, the application is still logging to the
log4J file, but now all future events are lost because the avro-client has

I thought the "-F" option to avro-client was like the "-F" option to tail,
but after looking at the code, it is not.  There seems to be no "follow"
mode for the avro client that I can see.  I then stumbled across this key
sentence from here<https://cwiki.apache.org/confluence/display/FLUME/Getting+Started#GettingStarted-flumengavroclientoptions>

 Think of the avro-client command as cat for Flume

So, it's a "cat", not a "tail".

So I'm wondering, what's the right/best/current way to emulate OG's

Much appreciated.
Stern, Mark 2012-08-13, 20:03
Patrick Wendell 2012-08-13, 23:28
Chris Neal 2012-08-14, 14:19
Chris Neal 2012-08-14, 15:00
Chris Neal 2012-08-14, 15:36
Patrick Wendell 2012-08-14, 20:15
Chris Neal 2012-08-15, 14:12