Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> File rollover


I am using hdfs sink. What I see is that when app is not in use I get 1000s
of 0 byte files and when I start to send out load flume is busy writing to
logs the following:

How can I improve this?

lume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672584.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672585.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672585.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672585.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672586.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672586.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672586.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672587.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672587.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672587.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672588.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672588.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672588.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672589.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672589.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672589.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672590.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672590.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672590.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672591.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672591.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672591.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672592.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672592.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672592.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672593.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672593.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672593.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672594.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672594.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672594.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672595.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672595.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672595.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672596.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672596.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672596.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672597.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672597.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672597.snappy
12/09/28 14:46:03 INFO hdfs.BucketWriter: Creating
maprfs:///user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672598.snappy.tmp
12/09/28 14:46:03 INFO hdfs.BucketWriter: Renaming
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas302/web.1348868672598.snappy.tmp
to
maprfs:/user/apuser/web-analytics/flume-snappy/2012/09/28/14/pprfdaaas