Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> File rollover


I am using hdfs sink. What I see is that when app is not in use I get 1000s
of 0 byte files and when I start to send out load flume is busy writing to
logs the following:

How can I improve this?

lume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672584.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672585.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672585.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672585.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672586.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672586.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672586.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672587.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672587.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672587.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672588.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672588.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672588.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672589.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672589.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672589.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672590.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672590.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672590.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672591.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672591.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672591.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672592.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672592.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672592.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672593.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672593.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672593.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672594.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672594.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672594.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672595.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672595.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672595.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672596.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672596.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672596.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672597.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672597.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672597.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672598.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672598.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB