Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - how to know which line the agent have sent ?


+
周梦想 2013-02-06, 02:45
Copy link to this message
-
Re: how to know which line the agent have sent ?
Alexander Alten-Lorenz 2013-02-06, 07:50
You haven't a control in such situations, since tailDir uses tail and holds the marker in memory.

We had few days ago a thread about:
http://search-hadoop.com/m/JV0lh2RDXLX/flume+tail+source+problem+and+performance&subj=flume+tail+source+problem+and+performance

- Alex

On Feb 6, 2013, at 3:45 AM, 周梦想 <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I'm using tailDirs('mydir') source of the agent to gather logs to hadoop
> hdfs. I notice some documents advise that if the agent collapsed,  I have
> to remove files in 'mydir' and clear flume.agent.logdir. Thus I will lose
> some data or have duplicate data. And I don't know which line the agent
> have sent to.
>
> I'm worrying about the agent failure and resend or miss-send the content to
> collector. I want to know how to check which line of log file the agent
> have sent if the agent exit suddenly. The files in flute log dir, such as
> sending,sent can't be read.
>
> Please give some advise to process such situation.
> Thanks.
>
> Andy Zhou

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
+
周梦想 2013-02-07, 03:03
+
周梦想 2013-02-07, 03:24
+
Jeong-shik Jang 2013-02-07, 04:18
+
周梦想 2013-02-07, 08:33
+
hoo.smth 2013-02-16, 09:00