Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Data in File-channel  data folder


Copy link to this message
-
Re: Data in File-channel data folder
Thank you Hari.

This is very useful information.

Let me try that scenario where I will send more data to create log-3.

Thanks

Madhu  Munagala
On Apr 11, 2013, at 2:57 PM, Hari Shreedharan <[EMAIL PROTECTED]> wrote:

> The File Channel deletes the files from which all events have been removed at the time of a checkpoint. But, the channel will keep 2 log files per data directory even if all its events are taken out. Once all events from log-1, log-2 are taken out and then events get written to log-3, log-1 will be deleted at the next checkpoint. Unless a log-3 is created, both log-1 and log-2 will not be deleted.
>
>
>
> Hari
>
> --
> Hari Shreedharan
>
> On Thursday, April 11, 2013 at 12:13 PM, Madhu Gmail wrote:
>
>> Hello,
>>
>> I have not heard from anyone. so just want make sure I have explained the issue correctly.
>>
>> I think this is a common problem for everyone who uses it flume.
>>
>> when flume sink consumes the log event from file channel, what will happen to the data that is committed to local disk under data directory.
>>
>> will it grow indefinitely like log-1, log-2, log-3.....and so on ???
>>
>> do I have to write script to remove the data from data directory ??
>>
>>
>>
>> Madhu Munagala
>> (214)679-2872
>>
>> On Apr 11, 2013, at 11:52 AM, Madhu Gmail <[EMAIL PROTECTED]> wrote:
>>
>>> Hello,
>>>
>>> How to clean up the data in file channel data folder. After the log events are processed by the sink, I still see the log-1 and log-2 shows 1.6GB and 1.2GB.
>>>
>>> once the log events are processed by the sink, the channel should not have any data in data directory under file-channel ....??
>>>
>>>
>>> Madhu Munagala
>>> (214)679-2872
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB