Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Why used space of flie channel buffer directory increase?


Copy link to this message
-
Re: Why used space of flie channel buffer directory increase?
Thanks for your reply.

I will try syslog as source.

Zhiwen Sun

On Wed, Mar 20, 2013 at 3:11 PM, Alexander Alten-Lorenz <[EMAIL PROTECTED]
> wrote:

> HI,
>
> I suspect tail -F and nc for filling up the directory. Whats inside of
> such a file which grows without a event?
>
> My assumption:
> nc is open one stream, and deliver over this stream all incoming events.
> Flume doesn't know that no event is coming in, since the stream never
> breaks up. I wondering if you could use syslog(-ng) for the event delivery?
>
> Cheers,
>  Alex
>
>
>
> On Mar 20, 2013, at 2:30 AM, Zhiwen Sun <[EMAIL PROTECTED]> wrote:
>
> > Thanks all for your reply.
> >
> > @Kenison
> > I stop my tail -F | nc program and there is no new event file in HDFS,
> so I think there is no event arrive. To make sure, I will test again with
> enable JMX.
> >
> > @Alex
> >
> > The latest log is following. I can't see any exception or warning.
> >
> > 13/03/19 15:28:16 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490901
> > 13/03/19 15:28:16 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp
> > 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 3
> > 13/03/19 15:28:17 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1363659953997, queueSize: 0,
> queueHead: 362981
> > 13/03/19 15:28:17 INFO file.LogFileV3: Updating log-7.meta
> currentPosition = 216278208, logWriteOrderID = 1363659953997
> > 13/03/19 15:28:17 INFO file.Log: Updated checkpoint for file:
> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216278208
> logWriteOrderID: 1363659953997
> > 13/03/19 15:28:26 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490902
> > 13/03/19 15:28:27 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp
> > 13/03/19 15:28:37 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490903
> > 13/03/19 15:28:37 INFO hdfs.BucketWriter: Creating hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp
> >
> > 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/zhiwensun/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 2
> > 13/03/19 15:28:47 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1363659954200, queueSize: 0,
> queueHead: 362981
> > 13/03/19 15:28:47 INFO file.LogFileV3: Updating log-7.meta
> currentPosition = 216288815, logWriteOrderID = 1363659954200
> > 13/03/19 15:28:47 INFO file.Log: Updated checkpoint for file:
> /home/zhiwensun/.flume/file-channel/data/log-7 position: 216288815
> logWriteOrderID: 1363659954200
> > 13/03/19 15:28:48 INFO hdfs.BucketWriter: Renaming hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904.tmp to hdfs://
> 127.0.0.1:9000/flume/events/2013-03-19/app.1363660490904
> >
> > @Hari
> > em, 12 hours passed. The size of file channel directory has no reduce.
> >
> > Files in file channel directory:
> >
> > -rw-r--r-- 1 zhiwensun zhiwensun    0 2013-03-19 09:15 in_use.lock
> > -rw-r--r-- 1 zhiwensun zhiwensun 1.0M 2013-03-19 10:11 log-6
> > -rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 10:12 log-6.meta
> > -rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28 log-7
> > -rw-r--r-- 1 zhiwensun zhiwensun   29 2013-03-19 15:28 log-7.meta
> > -rw-r--r-- 1 zhiwensun zhiwensun 207M 2013-03-19 15:28
> ./file-channel/data/log-7
> > -rw-r--r-- 1 zhiwensun zhiwensun 29 2013-03-19 10:12
> ./file-channel/data/log-6.meta
> > -rw-r--r-- 1 zhiwensun zhiwensun 29 2013-03-19 15:28