Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
Thanks for your reply.

I just wanna confirm whether the space of file channel has a limit.

Zhiwen Sun

On Wed, Mar 20, 2013 at 4:06 PM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

> If you reduce the capacity the channel will be able to buffer fewer
> events. If you want to reduce the space used when there are only a few
> events remaining set the config param: "maxFileSize" to something
> lower(this is in bytes). I don't advice setting this to lower than a few
> hundred megabytes (in fact, the default value works pretty well - do you
> really need to save 3GB space?)- else you will end up having a huge number
> of small files if there are many events wait to be taken from the channel.
>
>
> Hari
>
>
> On Wed, Mar 20, 2013 at 12:50 AM, Zhiwen Sun <[EMAIL PROTECTED]> wrote:
>
>> Hi Hari:
>>
>> Is that means I can reduce the capacity of file channel to cut down max
>> disk space used by file channel?
>>
>>
>> Zhiwen Sun
>>
>>
>>
>> On Wed, Mar 20, 2013 at 3:23 PM, Hari Shreedharan <
>> [EMAIL PROTECTED]> wrote:
>>
>>>  Hi,
>>>
>>> Like I mentioned earlier, we will always keep 2 data files in each data
>>> directory (the ".meta" files are metadata associated to the actual data).
>>> Once a log-8 is created(when log-7 gets rotated when it hits maximum size)
>>> and all of the events in log-6 are taken, then log-6 will get deleted, but
>>> you will still will see log-7 and log-8. So what you are seeing is not
>>> unexpected.
>>>
>>>
>>> Hari
>>>
>>> --
>>> Hari Shreedharan
>>>
>>> On Tuesday, March 19, 2013 at 6:30 PM, Zhiwen Sun wrote:
>>>
>>> Thanks all for your reply.
>>>
>>> @Kenison
>>> I stop my tail -F | nc program and there is no new event file in HDFS,
>>> so I think there is no event arrive. To make sure, I will test again with
>>> enable JMX.
>>>
>>> @Alex
>>>
>>> The latest log is following. I can't see any exception or warning.
>>>
>>> 13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp to hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901
>>> 13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp
>>> 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint
>>> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
>>> sync = 3
>>> 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating
>>> checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0,
>>> queueHead: 362981
>>> 13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta
>>> currentPosition = 216278208, logWriteOrderID = 1363659953997
>>> 13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file:
>>> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208
>>> logWriteOrderID: 1363659953997
>>> 13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp to hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902
>>> 13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp
>>> 13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp to hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903
>>> 13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://
>>> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp
>>>
>>> 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint
>>> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
>>> sync = 2
>>> 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating
>>> checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0,
>>> queueHead: 362981
>>> 13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta
>>> currentPosition = 216288815, logWriteOrderID = 1363659954200
>>> 13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file: