Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Proper documentation for setting up sink groups


Copy link to this message
-
Re: Proper documentation for setting up sink groups
Hi Hari,
Yes I did read the whole guide end to end.
But I still have doubts

The fact that multiple sinks can feed from the same channel is news to me.
I don't see it explicitly mentioned in the docs,
so i guess I assumed wrongly, that only one sink can feed from a channel.

a)Can you explain in detail , how having multiple sinks taking events from
one channel, is useful in a "fast source slow sinks" scenario ?

b) Also if I read your explanation below correctly there are 3 possible
cases

1) multiple sinks feeding from a single channel , with the default sink
processor this will be like a multiplexing channel with all sinks getting
all the events that come in the channel.

2) multiple sinks feeding from a single channel , with fail_over sink
processor, only one sink will get the events at a give time, with flume
failing over to next available sink in case the first one fails ?

3) multiple sinks feeding from a single channel, with
load balancing processor, with all sinks getting events in a
round-robin/random order.

Is this a correct assumption ? I am aware of #2 and #3, not sure about #1.

On Thu, Aug 23, 2012 at 12:43 PM, Hari Shreedharan <
[EMAIL PROTECTED]> wrote:

> Did you read this:
> http://flume.apache.org/FlumeUserGuide.html#flume-sink-processors
>
> That explains how to use sink groups. Also there is nothing wrong with
> multiple sinks taking events from one channel. This is an especially useful
> configuration if you have a very fast source and much slower sinks.
>
>
> Hari
>
> --
> Hari Shreedharan
>
>
> On Thursday, August 23, 2012 at 9:28 AM, Bhaskar V. Karambelkar wrote:
>
> > The sink group document doesn't mention anything about how
> > to hook up sink groups to the rest of the config in order to work.
> >
> > e.g. under normal circumstances one channel is linked with one sink.
> >
> > But for failover sink group , looks like both the sinks should be hooked
> up to the same channel,
> > but this is not mentioned any where.
> >
> > Similarly, what exactly needs to be done for load balancing sink ?
> >
> > thanks
>
>
>