Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


+
Zhiwen Sun 2013-03-19, 08:19
+
Kenison, Matt 2013-03-19, 16:27
+
Alexander Alten-Lorenz 2013-03-19, 17:32
+
Hari Shreedharan 2013-03-19, 18:32
+
Zhiwen Sun 2013-03-20, 01:30
+
Alexander Alten-Lorenz 2013-03-20, 07:11
+
Zhiwen Sun 2013-03-20, 07:15
Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
Hi,

Like I mentioned earlier, we will always keep 2 data files in each data directory (the ".meta" files are metadata associated to the actual data). Once a log-8 is created(when log-7 gets rotated when it hits maximum size) and all of the events in log-6 are taken, then log-6 will get deleted, but you will still will see log-7 and log-8. So what you are seeing is not unexpected.
Hari

--
Hari Shreedharan
On Tuesday, March 19, 2013 at 6:30 PM, Zhiwen Sun wrote:

> Thanks all for your reply.
>
> @Kenison
> I stop my tail -F | nc program and there is no new event file in HDFS, so I think there is no event arrive. To make sure, I will test again with enable JMX.
>
> @Alex
>
> The latest log is following. I can't see any exception or warning.
>
> > 13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp) to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901 (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901)
> > 13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp)
> > 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to sync = 3
> > 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0, queueHead: 362981
> > 13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta currentPosition = 216278208, logWriteOrderID = 1363659953997
> > 13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file: /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208 logWriteOrderID: 1363659953997
> > 13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp) to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902 (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902)
> > 13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp)
> > 13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp) to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903 (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903)
> > 13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp)
> >
> > 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to sync = 2
> > 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0, queueHead: 362981
> > 13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta currentPosition = 216288815, logWriteOrderID = 1363659954200
> > 13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file: /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815 logWriteOrderID: 1363659954200
> > 13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp) to hdfs://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904 (http://127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904)
>
> @Hari
> em, 12 hours passed. The size of file channel directory has no reduce.
>
> Files in file channel directory:
>
> > -rw-r--r-- 1 zhiwensun zhiwensun    0 2013-03-19 09:15 in_use.lock
+
Zhiwen Sun 2013-03-20, 07:50
+
Hari Shreedharan 2013-03-20, 08:06
+
Zhiwen Sun 2013-03-20, 09:20
+
Kenison, Matt 2013-03-20, 17:21
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB