Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
Hi,

Like I mentioned earlier, we will always keep 2 data files in each data directory (the ".meta" files are metadata associated to the actual data). Once a log-8 is created(when log-7 gets rotated when it hits maximum size) and all of the events in log-6 are taken, then log-6 will get deleted, but you will still will see log-7 and log-8. So what you are seeing is not unexpected.
Hari

--
Hari Shreedharan
On Tuesday, March 19, 2013 at 6:30 PM, Zhiwen Sun wrote:

> Thanks all for your reply.
>
> @Kenison
> I stop my tail -F | nc program and there is no new event file in HDFS, so I think there is no event arrive. To make sure, I will test again with enable JMX.
>
> @Alex
>
> The latest log is following. I can't see any exception or warning.
>
> > 13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp) to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901 (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901)
> > 13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp)
> > 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to sync = 3
> > 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0, queueHead: 362981
> > 13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta currentPosition = 216278208, logWriteOrderID = 1363659953997
> > 13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file: /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208 logWriteOrderID: 1363659953997
> > 13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp) to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902 (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902)
> > 13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp)
> > 13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp) to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903 (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903)
> > 13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp)
> >
> > 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to sync = 2
> > 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0, queueHead: 362981
> > 13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta currentPosition = 216288815, logWriteOrderID = 1363659954200
> > 13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file: /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815 logWriteOrderID: 1363659954200
> > 13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp) to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904 (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904)
>
> @Hari
> em, 12 hours passed. The size of file channel directory has no reduce.
>
> Files in file channel directory:
>
> > -rw-r--r-- 1 zhiwensun zhiwensun    0 2013-03-19 09:15 in_use.lock