Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


+
Zhiwen Sun 2013-03-19, 08:19
+
Kenison, Matt 2013-03-19, 16:27
+
Alexander Alten-Lorenz 2013-03-19, 17:32
+
Hari Shreedharan 2013-03-19, 18:32
+
Zhiwen Sun 2013-03-20, 01:30
+
Alexander Alten-Lorenz 2013-03-20, 07:11
+
Zhiwen Sun 2013-03-20, 07:15
+
Hari Shreedharan 2013-03-20, 07:23
+
Zhiwen Sun 2013-03-20, 07:50
+
Hari Shreedharan 2013-03-20, 08:06
+
Zhiwen Sun 2013-03-20, 09:20
Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
It is capped. You can verify this by using the stress source and a null sink. You'll see the disk usage increase to the maximum allowed and then plateau.
From: Zhiwen Sun <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Wed, 20 Mar 2013 02:20:53 -0700
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: Why used space of flie channel buffer directory increase?

Thanks for your reply.

I just wanna confirm whether the space of file channel has a limit.

Zhiwen Sun

On Wed, Mar 20, 2013 at 4:06 PM, Hari Shreedharan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
If you reduce the capacity the channel will be able to buffer fewer events. If you want to reduce the space used when there are only a few events remaining set the config param: "maxFileSize" to something lower(this is in bytes). I don't advice setting this to lower than a few hundred megabytes (in fact, the default value works pretty well - do you really need to save 3GB space?)- else you will end up having a huge number of small files if there are many events wait to be taken from the channel.
Hari
On Wed, Mar 20, 2013 at 12:50 AM, Zhiwen Sun <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Hari:

Is that means I can reduce the capacity of file channel to cut down max disk space used by file channel?
Zhiwen Sun

On Wed, Mar 20, 2013 at 3:23 PM, Hari Shreedharan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,

Like I mentioned earlier, we will always keep 2 data files in each data directory (the ".meta" files are metadata associated to the actual data). Once a log-8 is created(when log-7 gets rotated when it hits maximum size) and all of the events in log-6 are taken, then log-6 will get deleted, but you will still will see log-7 and log-8. So what you are seeing is not unexpected.
Hari

--
Hari Shreedharan
On Tuesday, March 19, 2013 at 6:30 PM, Zhiwen Sun wrote:

Thanks all for your reply.

@Kenison
I stop my tail -F | nc program and there is no new event file in HDFS, so I think there is no event arrive. To make sure, I will test again with enable JMX.

@Alex

The latest log is following. I can't see any exception or warning.

13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp> to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901>
13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp>
13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to sync = 3
13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0, queueHead: 362981
13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta currentPosition = 216278208, logWriteOrderID = 1363659953997
13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file: /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208 logWriteOrderID: 1363659953997
13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp> to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902>
13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp>
13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp> to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903>
13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp>

13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to sync = 2
13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0, queueHead: 362981
13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta currentPosition = 216288815, logWriteOrderID = 1363659954200
13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file: /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815 logWriteOrderID: 1363659954200
13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp> to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904>

@Hari
em, 12 hours passed. The size of file channel directory has no reduce.

Files in file channel directory:

-rw-r--r-- 1 zhiwensun zhiwensun    0 2013-03-19 09:15 in_use.lock
-rw-r--r-- 1 zhiwensun zhiwensun 1.0M 2013-03-19 10:11 log-6
-rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 10:12 log-6.meta
-rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28 log-7
-rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 15:28 log-7.meta
-rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28 ./file-channel/data/log-7
-rw-r--r-- 1 zhiwensun zhiwensun 29 2013-03-19 10:12 ./file-channel/data/log-6.me