Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> how to know which line the agent have sent ?


Copy link to this message
-
how to know which line the agent have sent ?
Hello,

I'm using tailDirs('mydir') source of the agent to gather logs to hadoop
hdfs. I notice some documents advise that if the agent collapsed,  I have
to remove files in 'mydir' and clear flume.agent.logdir. Thus I will lose
some data or have duplicate data. And I don't know which line the agent
have sent to.

I'm worrying about the agent failure and resend or miss-send the content to
collector. I want to know how to check which line of log file the agent
have sent if the agent exit suddenly. The files in flute log dir, such as
sending,sent can't be read.

Please give some advise to process such situation.
Thanks.

Andy Zhou
+
Alexander Alten-Lorenz 2013-02-06, 07:50
+
周梦想 2013-02-07, 03:03
+
周梦想 2013-02-07, 03:24
+
Jeong-shik Jang 2013-02-07, 04:18
+
周梦想 2013-02-07, 08:33
+
hoo.smth 2013-02-16, 09:00
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB