Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


+
Zhiwen Sun 2013-03-19, 08:19
+
Kenison, Matt 2013-03-19, 16:27
+
Alexander Alten-Lorenz 2013-03-19, 17:32
+
Hari Shreedharan 2013-03-19, 18:32
Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
Thanks all for your reply.

@Kenison
I stop my tail -F | nc program and there is no new event file in HDFS, so I
think there is no event arrive. To make sure, I will test again with enable
JMX.

@Alex

The latest log is following. I can't see any exception or warning.

13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901
> 13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp
> 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 3
> 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0,
> queueHead: 362981
> 13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 216278208, logWriteOrderID = 1363659953997
> 13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file:
> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208
> logWriteOrderID: 1363659953997
> 13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902
> 13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp
> 13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903
> 13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp
>
> 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 2
> 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0,
> queueHead: 362981
> 13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 216288815, logWriteOrderID = 1363659954200
> 13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file:
> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815
> logWriteOrderID: 1363659954200
> 13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904
>

@Hari
em, 12 hours passed. The size of file channel directory has no reduce.

Files in file channel directory:

-rw-r--r-- 1 zhiwensun zhiwensun    0 2013-03-19 09:15 in_use.lock
> -rw-r--r-- 1 zhiwensun zhiwensun 1.0M 2013-03-19 10:11 log-6
> -rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 10:12 log-6.meta
> -rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28 log-7
> -rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 15:28 log-7.meta
> -rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28
> ./file-channel/data/log-7
> -rw-r--r-- 1 zhiwensun zhiwensun 29 2013-03-19 10:12
> ./file-channel/data/log-6.meta
> -rw-r--r-- 1 zhiwensun zhiwensun 29 2013-03-19 15:28
> ./file-channel/data/log-7.meta
> -rw-r--r-- 1 zhiwensun zhiwensun 0 2013-03-19 09:15
> ./file-channel/data/in_use.lock
> -rw-r--r-- 1 zhiwensun zhiwensun 1.0M 2013-03-19 10:11
> ./file-channel/data/log-6
>

Zhiwen Sun

On Wed, Mar 20, 2013 at 2:32 AM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

>  It is possible for the directory size to increase even if no writes are
> going in to the channel. If the channel size is non-zero and the sink is
> still writing events to HDFS, the takes get written to disk as well (so we
> know what events in the files were removed when the channel/agent
> restarts). Eventually the channel will clean up the files which have all
> events taken (though it will keep at least 2 files per data directory, just
+
Alexander Alten-Lorenz 2013-03-20, 07:11
+
Zhiwen Sun 2013-03-20, 07:15
+
Hari Shreedharan 2013-03-20, 07:23
+
Zhiwen Sun 2013-03-20, 07:50
+
Hari Shreedharan 2013-03-20, 08:06
+
Zhiwen Sun 2013-03-20, 09:20
+
Kenison, Matt 2013-03-20, 17:21
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB