Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume File Channel Filling Up The Disk With Transaction Log, Any Way To Prevent It


Copy link to this message
-
Re: Flume File Channel Filling Up The Disk With Transaction Log, Any Way To Prevent It
Not sure what you mean by one agent each for event.
Its possible that you may be able to use one agent for your needs and this
could possibly alleviate disk contention by the agents as the file channel
is concerned.
Either way it sounds like you either need to decrease the size of the file
channel or increase the size of your disk.
On Mon, Nov 25, 2013 at 3:38 PM, Ritesh Adval <[EMAIL PROTECTED]>wrote:

> We have one agent each for event
> And metric and we have 3 hops where
> These goes through (rack, cluster and zone)  so we run these 2 agents
> together running on each hop. (total 6 agents, 2 in each VM)
>
> Is running single agent per VM recommend ?
>
> -Ritesh
>
>
>
> On Nov 25, 2013, at 3:23 PM, Jeff Lord <[EMAIL PROTECTED]> wrote:
>
> Its fine to run in a VM.
> Out of curiosity why are you running two agents on the machine though?
>
>
>
> On Mon, Nov 25, 2013 at 1:54 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> It the channel is full your clients will get a rejection notice.
>>
>> Capacity planning on the FC is a mix between event size, channel size,
>> and disk size. If flume is holding on to the logs, it's because it
>> needs them.  If you are constantly running out of space, then yes,
>> it's quite likely decreasing channel capacity is a logical course of
>> action.
>>
>> Brock
>>
>> On Mon, Nov 25, 2013 at 3:30 PM, Ritesh Adval <[EMAIL PROTECTED]>
>> wrote:
>> > Thanks but if it keeps any tx log which have events in channel, then it
>> > seems it would go out of diskspace, since our clients will keep sending
>> > events to it and it will keep creating those tx logs till it has
>> diskspace?
>> > Or Am I missing something here?
>> >
>> > what we need is the client to start getting meesage rejection if the
>> flume
>> > agent file channel has reached its limit in terms of pending messages
>> in tx
>> > logs or capacity.  Do you think we should reduce the channel capacity,
>> > currently it is set to 1M
>> >
>> >
>> > Ritesh
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Nov 25, 2013 at 1:00 PM, Brock Noland <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> It will keep any tx log that has a corresponding event in the channel
>> >> + 2 per data directory.
>> >>
>> >> On Mon, Nov 25, 2013 at 2:55 PM, Ritesh Adval <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Thanks but we do not know how many transaction log files it will
>> create,
>> >> > so
>> >> > it may go out of disk space even if we set lower maxFileSize.  Do we
>> >> > know
>> >> > how many max log files it will keep in flume 1.4 ?
>> >> >
>> >> > Ritesh
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Nov 25, 2013 at 12:50 PM, Brock Noland <[EMAIL PROTECTED]>
>> >> > wrote:
>> >> >>
>> >> >> Lower the maxFileSize.
>> >> >>
>> >> >> On Mon, Nov 25, 2013 at 2:41 PM, Ritesh Adval <
>> [EMAIL PROTECTED]>
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > We are running two flume 1.4  agents each with 2 file channel on
>> a VM
>> >> >> > of
>> >> >> > size 15GB.
>> >> >> >
>> >> >> > Is VM recommded to run flume or do we need bare metal boxes?
>> >> >> >
>> >> >> >
>> >> >> > Every week or so we are running into situation where due to our
>> sinks
>> >> >> > on
>> >> >> > these agents not able to send message to upstream agents, the
>> flume
>> >> >> > file
>> >> >> > channels get filled with large transaction logs.
>> >> >> >
>> >> >> > Here is what we see on 4 channels :
>> >> >> >
>> >> >> > $ du -h /srv/flume/
>> >> >> > 4.9G    /srv/flume/metricChannel1-Cluster/data
>> >> >> > 7.7M    /srv/flume/metricChannel1-Cluster/checkpoint
>> >> >> > 4.9G    /srv/flume/metricChannel1-Cluster
>> >> >> > 4.9G    /srv/flume/metricChannel2-Cluster/data
>> >> >> > 7.7M    /srv/flume/metricChannel2-Cluster/checkpoint
>> >> >> > 4.9G    /srv/flume/metricChannel2-Cluster
>> >> >> > 214M    /srv/flume/eventChannel2-Cluster/data
>> >> >> > 7.7M    /srv/flume/eventChannel2-Cluster/checkpoint
>> >> >> > 222M    /srv/flume/eventChannel2-Cluster
>> >> >> > 215M    /srv/flume/eventChannel1-Cluster/data
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB