Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume hdfs sink rollover


Copy link to this message
-
Flume hdfs sink rollover
I have rollover defined either to roll every 5G or 1+ hr but doesn't seem
to  be working. Could you please suggest if I got the conf incorrectly
configured?

foo.sinks.hdfsSink.hdfs.filePrefix = web
foo.sinks.hdfsSink.hdfs.rollInterval  = 4000
foo.sinks.hdfsSink.hdfs.rollCount  = 0
foo.sinks.hdfsSink.hdfs.rollSize  = 5000000000
foo.sinks.hdfsSink.hdfs.fileType  = SequenceFile
foo.sinks.hdfsSink.hdfs.codeC  = snappy

drwxr-xr-x   - root root          5 2012-08-24 14:58
/flume_vol/flume/2012/08/24/13/dslg1
-rwxr-xr-x   3 root root   28118740 2012-08-24 13:35
/flume_vol/flume/2012/08/24/13/dslg1/web.1345840475805.snappy.tmp
-rwxr-xr-x   3 root root  407700267 2012-08-24 13:57
/flume_vol/flume/2012/08/24/13/dslg1/web.1345840674872.snappy
-rwxr-xr-x   3 root root  407742601 2012-08-24 13:44
/flume_vol/flume/2012/08/24/13/dslg1/web.1345840674870.snappy
-rwxr-xr-x   3 root root  170657363 2012-08-24 14:58
/flume_vol/flume/2012/08/24/13/dslg1/web.1345840674873.snappy
-rwxr-xr-x   3 root root  407678663 2012-08-24 13:50
/flume_vol/flume/2012/08/24/13/dslg1/web.1345840674871.snappy
drwxr-xr-x   - root root          5 2012-08-24 15:04
/flume_vol/flume/2012/08/24/13/dslg2
-rwxr-xr-x   3 root root  407786389 2012-08-24 13:50
/flume_vol/flume/2012/08/24/13/dslg2/web.1345840668738.snappy
-rwxr-xr-x   3 root root  407757832 2012-08-24 13:44
/flume_vol/flume/2012/08/24/13/dslg2/web.1345840668737.snappy
-rwxr-xr-x   3 root root  159909773 2012-08-24 15:04
/flume_vol/flume/2012/08/24/13/dslg2/web.1345840668740.snappy
-rwxr-xr-x   3 root root   51085873 2012-08-24 13:36
/flume_vol/flume/2012/08/24/13/dslg2/web.1345840465501.snappy
-rwxr-xr-x   3 root root  407739053 2012-08-24 13:57
/flume_vol/flume/2012/08/24/13/dslg2/web.1345840668739.snappy
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB