Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Memory Channel


+
Mohit Anchlia 2013-01-16, 17:46
+
Hari Shreedharan 2013-01-16, 17:55
+
Mohit Anchlia 2013-01-16, 18:21
+
Brock Noland 2013-01-16, 18:26
+
Mohit Anchlia 2013-01-16, 18:47
+
Nitin Pawar 2013-01-16, 18:55
+
Brock Noland 2013-01-16, 19:00
Copy link to this message
-
Re: Memory Channel
Mohit Anchlia 2013-01-16, 19:03
Just one more question, when I write using memorychannel does that write
immediately gets written to the sink? It may not get sync on HDFS but does
it at least immediately gets written. I am trying to see if the events are
held in flume's memory or not.

On Wed, Jan 16, 2013 at 11:00 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

> The HDFS Sink syncs at the end of each batch or when the file rolls.
>
> On Wed, Jan 16, 2013 at 10:55 AM, Nitin Pawar <[EMAIL PROTECTED]>
> wrote:
> > you can configure it as you nee
> > number of events
> > rollover by time
> > and other ways as well
> >
> >
> > On Thu, Jan 17, 2013 at 12:17 AM, Mohit Anchlia <[EMAIL PROTECTED]>
> > wrote:
> >>
> >> Right. I was asking about sync to "sink". My sink is hdfs so does flume
> >> sync to hdfs on every write operation?
> >>
> >>
> >> On Wed, Jan 16, 2013 at 10:26 AM, Brock Noland <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>> Memory Channel does not write to disk and as such never syncs to disk.
> >>> File Channel does sync to disk for each batch put on or taken off the
> >>> channel.
> >>>
> >>> On Wed, Jan 16, 2013 at 10:21 AM, Mohit Anchlia <
> [EMAIL PROTECTED]>
> >>> wrote:
> >>> > Thanks! What I am really trying to understand is when does flume sync
> >>> > to the
> >>> > sink. I am not using batch events.
> >>> >
> >>> >
> >>> > On Wed, Jan 16, 2013 at 9:55 AM, Hari Shreedharan
> >>> > <[EMAIL PROTECTED]> wrote:
> >>> >>
> >>> >> It means that the channel can store that many events. If it is full,
> >>> >> then
> >>> >> the put() calls (on the source side) will start throwing
> >>> >> ChannelException.
> >>> >> The put call will block only for keep-alive number of seconds, after
> >>> >> which
> >>> >> it will throw.
> >>> >>
> >>> >>
> >>> >> Hari
> >>> >>
> >>> >> --
> >>> >> Hari Shreedharan
> >>> >>
> >>> >> On Wednesday, January 16, 2013 at 9:46 AM, Mohit Anchlia wrote:
> >>> >>
> >>> >> Could someone help me understand capacity attribute of
> memoryChannel?
> >>> >> Does
> >>> >> it mean that memoryChannel flushes to sink only when this capacity
> is
> >>> >> reached or does it mean that it's the max events stored in memory
> and
> >>> >> call
> >>> >> blocks until everything else gets freed?
> >>> >>
> >>> >>
> >>> >> http://flume.apache.org/FlumeUserGuide.html#memory-channel
> >>> >>
> >>> >>
> >>> >>
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Apache MRUnit - Unit testing MapReduce -
> >>> http://incubator.apache.org/mrunit/
> >>
> >>
> >
> >
> >
> > --
> > Nitin Pawar
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
+
Roshan Naik 2013-01-17, 01:13
+
Juhani Connolly 2013-01-17, 02:00