Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Data in File-channel  data folder


+
Madhu Gmail 2013-04-11, 16:52
+
Madhu Gmail 2013-04-11, 19:13
Copy link to this message
-
Re: Data in File-channel data folder
The File Channel deletes the files from which all events have been removed at the time of a checkpoint. But, the channel will keep 2 log files per data directory even if all its events are taken out. Once all events from log-1, log-2 are taken out and then events get written to log-3, log-1 will be deleted at the next checkpoint. Unless a log-3 is created, both log-1 and log-2 will not be deleted.

Hari

--
Hari Shreedharan
On Thursday, April 11, 2013 at 12:13 PM, Madhu Gmail wrote:

> Hello,
>
> I have not heard from anyone. so just want make sure I have explained the issue correctly.
>
> I think this is a common problem for everyone who uses it flume.
>
> when flume sink consumes the log event from file channel, what will happen to the data that is committed to local disk under data directory.
>
> will it grow indefinitely like log-1, log-2, log-3.....and so on ???
>
> do I have to write script to remove the data from data directory ??
>
>
>
> Madhu Munagala
> (214)679-2872
>
> On Apr 11, 2013, at 11:52 AM, Madhu Gmail <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > Hello,
> >
> > How to clean up the data in file channel data folder. After the log events are processed by the sink, I still see the log-1 and log-2 shows 1.6GB and 1.2GB.
> >
> > once the log events are processed by the sink, the channel should not have any data in data directory under file-channel ....??
> >
> >
> > Madhu Munagala
> > (214)679-2872
> >
>
>
>
+
Madhu Gmail 2013-04-11, 20:11
+
Mike Keane 2013-04-11, 20:08
+
Madhu Gmail 2013-04-11, 20:17
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB