Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> how to know which line the agent have sent ?


Copy link to this message
-
Re: how to know which line the agent have sent ?
You haven't a control in such situations, since tailDir uses tail and holds the marker in memory.

We had few days ago a thread about:
http://search-hadoop.com/m/JV0lh2RDXLX/flume+tail+source+problem+and+performance&subj=flume+tail+source+problem+and+performance

- Alex

On Feb 6, 2013, at 3:45 AM, 周梦想 <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I'm using tailDirs('mydir') source of the agent to gather logs to hadoop
> hdfs. I notice some documents advise that if the agent collapsed,  I have
> to remove files in 'mydir' and clear flume.agent.logdir. Thus I will lose
> some data or have duplicate data. And I don't know which line the agent
> have sent to.
>
> I'm worrying about the agent failure and resend or miss-send the content to
> collector. I want to know how to check which line of log file the agent
> have sent if the agent exit suddenly. The files in flute log dir, such as
> sending,sent can't be read.
>
> Please give some advise to process such situation.
> Thanks.
>
> Andy Zhou

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB