Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
It is capped. You can verify this by using the stress source and a null sink. You'll see the disk usage increase to the maximum allowed and then plateau.
From: Zhiwen Sun <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Wed, 20 Mar 2013 02:20:53 -0700
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: Why used space of flie channel buffer directory increase?

Thanks for your reply.

I just wanna confirm whether the space of file channel has a limit.

Zhiwen Sun

On Wed, Mar 20, 2013 at 4:06 PM, Hari Shreedharan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
If you reduce the capacity the channel will be able to buffer fewer events. If you want to reduce the space used when there are only a few events remaining set the config param: "maxFileSize" to something lower(this is in bytes). I don't advice setting this to lower than a few hundred megabytes (in fact, the default value works pretty well - do you really need to save 3GB space?)- else you will end up having a huge number of small files if there are many events wait to be taken from the channel.
Hari
On Wed, Mar 20, 2013 at 12:50 AM, Zhiwen Sun <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Hari:

Is that means I can reduce the capacity of file channel to cut down max disk space used by file channel?
Zhiwen Sun

On Wed, Mar 20, 2013 at 3:23 PM, Hari Shreedharan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,

Like I mentioned earlier, we will always keep 2 data files in each data directory (the ".meta" files are metadata associated to the actual data). Once a log-8 is created(when log-7 gets rotated when it hits maximum size) and all of the events in log-6 are taken, then log-6 will get deleted, but you will still will see log-7 and log-8. So what you are seeing is not unexpected.
Hari

--
Hari Shreedharan
On Tuesday, March 19, 2013 at 6:30 PM, Zhiwen Sun wrote:

Thanks all for your reply.

@Kenison
I stop my tail -F | nc program and there is no new event file in HDFS, so I think there is no event arrive. To make sure, I will test again with enable JMX.

@Alex

The latest log is following. I can't see any exception or warning.

13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp> to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901>
13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp>
13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to sync = 3
13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0, queueHead: 362981
13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta currentPosition = 216278208, logWriteOrderID = 1363659953997
13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file: /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208 logWriteOrderID: 1363659953997
13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp> to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902>
13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp>
13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp> to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903>
13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp>

13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to sync = 2
13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0, queueHead: 362981
13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta currentPosition = 216288815, logWriteOrderID = 1363659954200
13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file: /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815 logWriteOrderID: 1363659954200
13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp> to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904<http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904>

@Hari
em, 12 hours passed. The size of file channel directory has no reduce.

Files in file channel directory:

-rw-r--r-- 1 zhiwensun zhiwensun    0 2013-03-19 09:15 in_use.lock
-rw-r--r-- 1 zhiwensun zhiwensun 1.0M 2013-03-19 10:11 log-6
-rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 10:12 log-6.meta
-rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28 log-7
-rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 15:28 log-7.meta
-rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28 ./file-channel/data/log-7
-rw-r--r-- 1 zhiwensun zhiwensun 29 2013-03-19 10:12 ./file-channel/data/log-6.me
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB