Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Memory Channel


Copy link to this message
-
Re: Memory Channel
The HDFS Sink syncs at the end of each batch or when the file rolls.

On Wed, Jan 16, 2013 at 10:55 AM, Nitin Pawar <[EMAIL PROTECTED]> wrote:
> you can configure it as you nee
> number of events
> rollover by time
> and other ways as well
>
>
> On Thu, Jan 17, 2013 at 12:17 AM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
>>
>> Right. I was asking about sync to "sink". My sink is hdfs so does flume
>> sync to hdfs on every write operation?
>>
>>
>> On Wed, Jan 16, 2013 at 10:26 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>>
>>> Memory Channel does not write to disk and as such never syncs to disk.
>>> File Channel does sync to disk for each batch put on or taken off the
>>> channel.
>>>
>>> On Wed, Jan 16, 2013 at 10:21 AM, Mohit Anchlia <[EMAIL PROTECTED]>
>>> wrote:
>>> > Thanks! What I am really trying to understand is when does flume sync
>>> > to the
>>> > sink. I am not using batch events.
>>> >
>>> >
>>> > On Wed, Jan 16, 2013 at 9:55 AM, Hari Shreedharan
>>> > <[EMAIL PROTECTED]> wrote:
>>> >>
>>> >> It means that the channel can store that many events. If it is full,
>>> >> then
>>> >> the put() calls (on the source side) will start throwing
>>> >> ChannelException.
>>> >> The put call will block only for keep-alive number of seconds, after
>>> >> which
>>> >> it will throw.
>>> >>
>>> >>
>>> >> Hari
>>> >>
>>> >> --
>>> >> Hari Shreedharan
>>> >>
>>> >> On Wednesday, January 16, 2013 at 9:46 AM, Mohit Anchlia wrote:
>>> >>
>>> >> Could someone help me understand capacity attribute of memoryChannel?
>>> >> Does
>>> >> it mean that memoryChannel flushes to sink only when this capacity is
>>> >> reached or does it mean that it's the max events stored in memory and
>>> >> call
>>> >> blocks until everything else gets freed?
>>> >>
>>> >>
>>> >> http://flume.apache.org/FlumeUserGuide.html#memory-channel
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>
>>
>
>
>
> --
> Nitin Pawar

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/