Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Why used space of flie channel buffer directory increase?


Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
Hari Shreedharan 2013-03-20, 08:06
If you reduce the capacity the channel will be able to buffer fewer events.
If you want to reduce the space used when there are only a few events
remaining set the config param: "maxFileSize" to something lower(this is in
bytes). I don't advice setting this to lower than a few hundred megabytes
(in fact, the default value works pretty well - do you really need to save
3GB space?)- else you will end up having a huge number of small files if
there are many events wait to be taken from the channel.
Hari
On Wed, Mar 20, 2013 at 12:50 AM, Zhiwen Sun <[EMAIL PROTECTED]> wrote:

> Hi Hari:
>
> Is that means I can reduce the capacity of file channel to cut down max
> disk space used by file channel?
>
>
> Zhiwen Sun
>
>
>
> On Wed, Mar 20, 2013 at 3:23 PM, Hari Shreedharan <
> [EMAIL PROTECTED]> wrote:
>
>>  Hi,
>>
>> Like I mentioned earlier, we will always keep 2 data files in each data
>> directory (the ".meta" files are metadata associated to the actual data).
>> Once a log-8 is created(when log-7 gets rotated when it hits maximum size)
>> and all of the events in log-6 are taken, then log-6 will get deleted, but
>> you will still will see log-7 and log-8. So what you are seeing is not
>> unexpected.
>>
>>
>> Hari
>>
>> --
>> Hari Shreedharan
>>
>> On Tuesday, March 19, 2013 at 6:30 PM, Zhiwen Sun wrote:
>>
>> Thanks all for your reply.
>>
>> @Kenison
>> I stop my tail -F | nc program and there is no new event file in HDFS, so
>> I think there is no event arrive. To make sure, I will test again with
>> enable JMX.
>>
>> @Alex
>>
>> The latest log is following. I can't see any exception or warning.
>>
>> 13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp to hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901
>> 13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp
>> 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint
>> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
>> sync = 3
>> 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating
>> checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0,
>> queueHead: 362981
>> 13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta
>> currentPosition = 216278208, logWriteOrderID = 1363659953997
>> 13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file:
>> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208
>> logWriteOrderID: 1363659953997
>> 13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp to hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902
>> 13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp
>> 13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp to hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903
>> 13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp
>>
>> 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint
>> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
>> sync = 2
>> 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating
>> checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0,
>> queueHead: 362981
>> 13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta
>> currentPosition = 216288815, logWriteOrderID = 1363659954200
>> 13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file:
>> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815
>> logWriteOrderID: 1363659954200
>> 13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp to hdfs://
>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904