I am not sure if there is a simple and perfect solution for both loss
and duplication at failure using Flume or other.
for example with Flume-OG,
using E2E reliability mode, you can minimize loss but duplication can
happen; using BE mode with startFromEnd=true for tail, you can minimize
duplication but loss can happen.
At this moment, we are using combination of our own plug-ins to minimize
the affect at failure and monitoring/alert system to response quickly.
On 2/7/13 12:24 PM, 锟斤拷锟斤拷锟斤拷 wrote:
> So all users of flume don't care the agent break down and miss send or
> duplicate the content of logs? They have to write their own sources
> and sinks?
> They don't care the correct of logs? How they do if the flume agent
> I'm not yet understand.
> 2013/2/7 锟斤拷锟斤拷锟斤拷 <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> I see锟斤拷there is no easy way or configure way to know the detail of
> what has sent and what haven't.
> I have to write my own source or sink code to do this.
> Thank you,Alex and all friends.
> 2013/2/6 Alexander Alten-Lorenz <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>>
> You haven't a control in such situations, since tailDir uses
> tail and holds the marker in memory.
> We had few days ago a thread about:
> - Alex
> On Feb 6, 2013, at 3:45 AM, 锟斤拷锟斤拷锟斤拷 <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
> > Hello,
> > I'm using tailDirs('mydir') source of the agent to gather
> logs to hadoop
> > hdfs. I notice some documents advise that if the agent
> collapsed, I have
> > to remove files in 'mydir' and clear flume.agent.logdir.
> Thus I will lose
> > some data or have duplicate data. And I don't know which
> line the agent
> > have sent to.
> > I'm worrying about the agent failure and resend or miss-send
> the content to
> > collector. I want to know how to check which line of log
> file the agent
> > have sent if the agent exit suddenly. The files in flute log
> dir, such as
> > sending,sent can't be read.
> > Please give some advise to process such situation.
> > Thanks.
> > Andy Zhou
> Alexander Alten-Lorenz
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader