Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Flume File Channel Filling Up The Disk With Transaction Log, Any Way To Prevent It


+
Ritesh Adval 2013-11-25, 20:41
+
Brock Noland 2013-11-25, 20:50
+
Ritesh Adval 2013-11-25, 20:55
+
Brock Noland 2013-11-25, 21:00
+
Ritesh Adval 2013-11-25, 21:30
+
Brock Noland 2013-11-25, 21:54
Copy link to this message
-
Re: Flume File Channel Filling Up The Disk With Transaction Log, Any Way To Prevent It
Jeff Lord 2013-11-25, 23:23
Its fine to run in a VM.
Out of curiosity why are you running two agents on the machine though?

On Mon, Nov 25, 2013 at 1:54 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> It the channel is full your clients will get a rejection notice.
>
> Capacity planning on the FC is a mix between event size, channel size,
> and disk size. If flume is holding on to the logs, it's because it
> needs them.  If you are constantly running out of space, then yes,
> it's quite likely decreasing channel capacity is a logical course of
> action.
>
> Brock
>
> On Mon, Nov 25, 2013 at 3:30 PM, Ritesh Adval <[EMAIL PROTECTED]>
> wrote:
> > Thanks but if it keeps any tx log which have events in channel, then it
> > seems it would go out of diskspace, since our clients will keep sending
> > events to it and it will keep creating those tx logs till it has
> diskspace?
> > Or Am I missing something here?
> >
> > what we need is the client to start getting meesage rejection if the
> flume
> > agent file channel has reached its limit in terms of pending messages in
> tx
> > logs or capacity.  Do you think we should reduce the channel capacity,
> > currently it is set to 1M
> >
> >
> > Ritesh
> >
> >
> >
> >
> >
> >
> > On Mon, Nov 25, 2013 at 1:00 PM, Brock Noland <[EMAIL PROTECTED]>
> wrote:
> >>
> >> It will keep any tx log that has a corresponding event in the channel
> >> + 2 per data directory.
> >>
> >> On Mon, Nov 25, 2013 at 2:55 PM, Ritesh Adval <[EMAIL PROTECTED]>
> >> wrote:
> >> > Thanks but we do not know how many transaction log files it will
> create,
> >> > so
> >> > it may go out of disk space even if we set lower maxFileSize.  Do we
> >> > know
> >> > how many max log files it will keep in flume 1.4 ?
> >> >
> >> > Ritesh
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, Nov 25, 2013 at 12:50 PM, Brock Noland <[EMAIL PROTECTED]>
> >> > wrote:
> >> >>
> >> >> Lower the maxFileSize.
> >> >>
> >> >> On Mon, Nov 25, 2013 at 2:41 PM, Ritesh Adval <
> [EMAIL PROTECTED]>
> >> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > We are running two flume 1.4  agents each with 2 file channel on a
> VM
> >> >> > of
> >> >> > size 15GB.
> >> >> >
> >> >> > Is VM recommded to run flume or do we need bare metal boxes?
> >> >> >
> >> >> >
> >> >> > Every week or so we are running into situation where due to our
> sinks
> >> >> > on
> >> >> > these agents not able to send message to upstream agents, the flume
> >> >> > file
> >> >> > channels get filled with large transaction logs.
> >> >> >
> >> >> > Here is what we see on 4 channels :
> >> >> >
> >> >> > $ du -h /srv/flume/
> >> >> > 4.9G    /srv/flume/metricChannel1-Cluster/data
> >> >> > 7.7M    /srv/flume/metricChannel1-Cluster/checkpoint
> >> >> > 4.9G    /srv/flume/metricChannel1-Cluster
> >> >> > 4.9G    /srv/flume/metricChannel2-Cluster/data
> >> >> > 7.7M    /srv/flume/metricChannel2-Cluster/checkpoint
> >> >> > 4.9G    /srv/flume/metricChannel2-Cluster
> >> >> > 214M    /srv/flume/eventChannel2-Cluster/data
> >> >> > 7.7M    /srv/flume/eventChannel2-Cluster/checkpoint
> >> >> > 222M    /srv/flume/eventChannel2-Cluster
> >> >> > 215M    /srv/flume/eventChannel1-Cluster/data
> >> >> > 7.7M    /srv/flume/eventChannel1-Cluster/checkpoint
> >> >> > 223M    /srv/flume/eventChannel1-Cluster
> >> >> > 11G     /srv/flume/
> >> >> >
> >> >> >
> >> >> > Here is an example of tx logs on metricChannel1, we are seeing 5
> log
> >> >> > files.
> >> >> > Is there
> >> >> > a way to restrict the number of log files kept? I think in older
> >> >> > version
> >> >> > of
> >> >> > flume it was max 2 log files but we are seeing more than 2 as shown
> >> >> > below:
> >> >> >
> >> >> >
> >> >> >  $ ls -l /srv/flume/metricChannel1-Cluster/data/
> >> >> > total 4.5G
> >> >> > -rw-r--r-- 1 flume flume    0 Nov 23 00:39 in_use.lock
> >> >> > -rw-r--r-- 1 flume flume 1.1G Nov 23 11:11 log-1
> >> >> > -rw-r--r-- 1 flume flume   47 Nov 24 21:14 log-1.meta
> >> >> > -rw-r--r-- 1 flume flume 1.1G Nov 23 21:18 log-2
> >
+
Ritesh Adval 2013-11-25, 23:38
+
Jeff Lord 2013-11-26, 01:04
+
Ritesh Adval 2013-11-26, 01:30