Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
If you reduce the capacity the channel will be able to buffer fewer events.
If you want to reduce the space used when there are only a few events
remaining set the config param: "maxFileSize" to something lower(this is in
bytes). I don't advice setting this to lower than a few hundred megabytes
(in fact, the default value works pretty well - do you really need to save
3GB space?)- else you will end up having a huge number of small files if
there are many events wait to be taken from the channel.
Hari
On Wed, Mar 20, 2013 at 12:50 AM, Zhiwen Sun <[EMAIL PROTECTED]> wrote:

> Hi Hari:
>
> Is that means I can reduce the capacity of file channel to cut down max
> disk space used by file channel?
>
>
> Zhiwen Sun
>
>
>
> On Wed, Mar 20, 2013 at 3:23 PM, Hari Shreedharan <
> [EMAIL PROTECTED]> wrote:
>
>>  Hi,
>>
>> Like I mentioned earlier, we will always keep 2 data files in each data
>> directory (the ".meta" files are metadata associated to the actual data).
>> Once a log-8 is created(when log-7 gets rotated when it hits maximum size)
>> and all of the events in log-6 are taken, then log-6 will get deleted, but
>> you will still will see log-7 and log-8. So what you are seeing is not
>> unexpected.
>>
>>
>> Hari
>>
>> --
>> Hari Shreedharan
>>
>> On Tuesday, March 19, 2013 at 6:30 PM, Zhiwen Sun wrote:
>>
>> Thanks all for your reply.
>>
>> @Kenison
>> I stop my tail -F | nc program and there is no new event file in HDFS, so
>> I think there is no event arrive. To make sure, I will test again with
>> enable JMX.
>>
>> @Alex
>>
>> The latest log is following. I can't see any exception or warning.
>>
>> 13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp to hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901
>> 13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp
>> 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint
>> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
>> sync = 3
>> 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating
>> checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0,
>> queueHead: 362981
>> 13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta
>> currentPosition = 216278208, logWriteOrderID = 1363659953997
>> 13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file:
>> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208
>> logWriteOrderID: 1363659953997
>> 13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp to hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902
>> 13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp
>> 13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp to hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903
>> 13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp
>>
>> 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint
>> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
>> sync = 2
>> 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating
>> checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0,
>> queueHead: 362981
>> 13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta
>> currentPosition = 216288815, logWriteOrderID = 1363659954200
>> 13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file:
>> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815
>> logWriteOrderID: 1363659954200
>> 13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp to hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904