Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume Sink Cleanup


Copy link to this message
-
Re: Flume Sink Cleanup
Hi Rajesh,

You have to write a cron job, a overwrite after some time isn't possible.

Regards,
 Alex

On Jun 20, 2013, at 4:59 PM, Rajesh Jain <[EMAIL PROTECTED]> wrote:

> Hi
>
> This is a newbie question.
>
> I have configured flume sink with the Rolling File Sink
>
> I have the following in my flume-conf.properties page
>
> agent1.sinks.purepath.type = com.x.diagnostics.flume.RollingFileSink
> # once an hour
> agent1.sinks.purepath.sink.rollInterval = 3600
> # Force cutoff at 100 MB
> agent1.sinks.purepath.sink.rollSize = 100
> agent1.sinks.purepath.sink.directory = /log/export
> agent1.sinks.purepath.batchSize = 1000
>
> The directory fills up and is there a way to clean it up. Do I have to write an external cron job to clean up the directory, or is there a way to overwrite the files (recycle) after a specified period - say 6 months
>
> Thanks, Rajesh
>

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB