Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> how to know which line the agent have sent ?


Copy link to this message
-
how to know which line the agent have sent ?
Hello,

I'm using tailDirs('mydir') source of the agent to gather logs to hadoop
hdfs. I notice some documents advise that if the agent collapsed,  I have
to remove files in 'mydir' and clear flume.agent.logdir. Thus I will lose
some data or have duplicate data. And I don't know which line the agent
have sent to.

I'm worrying about the agent failure and resend or miss-send the content to
collector. I want to know how to check which line of log file the agent
have sent if the agent exit suddenly. The files in flute log dir, such as
sending,sent can't be read.

Please give some advise to process such situation.
Thanks.

Andy Zhou