Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Memory Channel


Copy link to this message
-
Re: Memory Channel
Just one more question, when I write using memorychannel does that write
immediately gets written to the sink? It may not get sync on HDFS but does
it at least immediately gets written. I am trying to see if the events are
held in flume's memory or not.

On Wed, Jan 16, 2013 at 11:00 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

> The HDFS Sink syncs at the end of each batch or when the file rolls.
>
> On Wed, Jan 16, 2013 at 10:55 AM, Nitin Pawar <[EMAIL PROTECTED]>
> wrote:
> > you can configure it as you nee
> > number of events
> > rollover by time
> > and other ways as well
> >
> >
> > On Thu, Jan 17, 2013 at 12:17 AM, Mohit Anchlia <[EMAIL PROTECTED]>
> > wrote:
> >>
> >> Right. I was asking about sync to "sink". My sink is hdfs so does flume
> >> sync to hdfs on every write operation?
> >>
> >>
> >> On Wed, Jan 16, 2013 at 10:26 AM, Brock Noland <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>> Memory Channel does not write to disk and as such never syncs to disk.
> >>> File Channel does sync to disk for each batch put on or taken off the
> >>> channel.
> >>>
> >>> On Wed, Jan 16, 2013 at 10:21 AM, Mohit Anchlia <
> [EMAIL PROTECTED]>
> >>> wrote:
> >>> > Thanks! What I am really trying to understand is when does flume sync
> >>> > to the
> >>> > sink. I am not using batch events.
> >>> >
> >>> >
> >>> > On Wed, Jan 16, 2013 at 9:55 AM, Hari Shreedharan
> >>> > <[EMAIL PROTECTED]> wrote:
> >>> >>
> >>> >> It means that the channel can store that many events. If it is full,
> >>> >> then
> >>> >> the put() calls (on the source side) will start throwing
> >>> >> ChannelException.
> >>> >> The put call will block only for keep-alive number of seconds, after
> >>> >> which
> >>> >> it will throw.
> >>> >>
> >>> >>
> >>> >> Hari
> >>> >>
> >>> >> --
> >>> >> Hari Shreedharan
> >>> >>
> >>> >> On Wednesday, January 16, 2013 at 9:46 AM, Mohit Anchlia wrote:
> >>> >>
> >>> >> Could someone help me understand capacity attribute of
> memoryChannel?
> >>> >> Does
> >>> >> it mean that memoryChannel flushes to sink only when this capacity
> is
> >>> >> reached or does it mean that it's the max events stored in memory
> and
> >>> >> call
> >>> >> blocks until everything else gets freed?
> >>> >>
> >>> >>
> >>> >> http://flume.apache.org/FlumeUserGuide.html#memory-channel
> >>> >>
> >>> >>
> >>> >>
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Apache MRUnit - Unit testing MapReduce -
> >>> http://incubator.apache.org/mrunit/
> >>
> >>
> >
> >
> >
> > --
> > Nitin Pawar
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB