Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
Thanks for your reply.

I will try syslog as source.

Zhiwen Sun

On Wed, Mar 20, 2013 at 3:11 PM, Alexander Alten-Lorenz <[EMAIL PROTECTED]
> wrote:

> HI,
>
> I suspect tail -F and nc for filling up the directory. Whats inside of
> such a file which grows without a event?
>
> My assumption:
> nc is open one stream, and deliver over this stream all incoming events.
> Flume doesn't know that no event is coming in, since the stream never
> breaks up. I wondering if you could use syslog(-ng) for the event delivery?
>
> Cheers,
>  Alex
>
>
>
> On Mar 20, 2013, at 2:30 AM, Zhiwen Sun <[EMAIL PROTECTED]> wrote:
>
> > Thanks all for your reply.
> >
> > @Kenison
> > I stop my tail -F | nc program and there is no new event file in HDFS,
> so I think there is no event arrive. To make sure, I will test again with
> enable JMX.
> >
> > @Alex
> >
> > The latest log is following. I can't see any exception or warning.
> >
> > 13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901
> > 13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp
> > 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 3
> > 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0,
> queueHead: 362981
> > 13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta
> currentPosition = 216278208, logWriteOrderID = 1363659953997
> > 13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file:
> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208
> logWriteOrderID: 1363659953997
> > 13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902
> > 13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp
> > 13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903
> > 13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp
> >
> > 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 2
> > 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0,
> queueHead: 362981
> > 13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta
> currentPosition = 216288815, logWriteOrderID = 1363659954200
> > 13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file:
> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815
> logWriteOrderID: 1363659954200
> > 13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904
> >
> > @Hari
> > em, 12 hours passed. The size of file channel directory has no reduce.
> >
> > Files in file channel directory:
> >
> > -rw-r--r-- 1 zhiwensun zhiwensun    0 2013-03-19 09:15 in_use.lock
> > -rw-r--r-- 1 zhiwensun zhiwensun 1.0M 2013-03-19 10:11 log-6
> > -rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 10:12 log-6.meta
> > -rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28 log-7
> > -rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 15:28 log-7.meta
> > -rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28
> ./file-channel/data/log-7
> > -rw-r--r-- 1 zhiwensun zhiwensun 29 2013-03-19 10:12
> ./file-channel/data/log-6.meta
> > -rw-r--r-- 1 zhiwensun zhiwensun 29 2013-03-19 15:28
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB