Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - how to know which line the agent have sent ?

Copy link to this message
how to know which line the agent have sent ?
周梦想 2013-02-06, 02:45

I'm using tailDirs('mydir') source of the agent to gather logs to hadoop
hdfs. I notice some documents advise that if the agent collapsed,  I have
to remove files in 'mydir' and clear flume.agent.logdir. Thus I will lose
some data or have duplicate data. And I don't know which line the agent
have sent to.

I'm worrying about the agent failure and resend or miss-send the content to
collector. I want to know how to check which line of log file the agent
have sent if the agent exit suddenly. The files in flute log dir, such as
sending,sent can't be read.

Please give some advise to process such situation.

Andy Zhou
Alexander Alten-Lorenz 2013-02-06, 07:50
周梦想 2013-02-07, 03:03
周梦想 2013-02-07, 03:24
Jeong-shik Jang 2013-02-07, 04:18
周梦想 2013-02-07, 08:33
hoo.smth 2013-02-16, 09:00