Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Proper documentation for setting up sink groups


+
Bhaskar V. Karambelkar 2012-08-23, 16:28
+
Hari Shreedharan 2012-08-23, 16:43
Copy link to this message
-
Re: Proper documentation for setting up sink groups
Hi Hari,
Yes I did read the whole guide end to end.
But I still have doubts

The fact that multiple sinks can feed from the same channel is news to me.
I don't see it explicitly mentioned in the docs,
so i guess I assumed wrongly, that only one sink can feed from a channel.

a)Can you explain in detail , how having multiple sinks taking events from
one channel, is useful in a "fast source slow sinks" scenario ?

b) Also if I read your explanation below correctly there are 3 possible
cases

1) multiple sinks feeding from a single channel , with the default sink
processor this will be like a multiplexing channel with all sinks getting
all the events that come in the channel.

2) multiple sinks feeding from a single channel , with fail_over sink
processor, only one sink will get the events at a give time, with flume
failing over to next available sink in case the first one fails ?

3) multiple sinks feeding from a single channel, with
load balancing processor, with all sinks getting events in a
round-robin/random order.

Is this a correct assumption ? I am aware of #2 and #3, not sure about #1.

On Thu, Aug 23, 2012 at 12:43 PM, Hari Shreedharan <
[EMAIL PROTECTED]> wrote:

> Did you read this:
> http://flume.apache.org/FlumeUserGuide.html#flume-sink-processors
>
> That explains how to use sink groups. Also there is nothing wrong with
> multiple sinks taking events from one channel. This is an especially useful
> configuration if you have a very fast source and much slower sinks.
>
>
> Hari
>
> --
> Hari Shreedharan
>
>
> On Thursday, August 23, 2012 at 9:28 AM, Bhaskar V. Karambelkar wrote:
>
> > The sink group document doesn't mention anything about how
> > to hook up sink groups to the rest of the config in order to work.
> >
> > e.g. under normal circumstances one channel is linked with one sink.
> >
> > But for failover sink group , looks like both the sinks should be hooked
> up to the same channel,
> > but this is not mentioned any where.
> >
> > Similarly, what exactly needs to be done for load balancing sink ?
> >
> > thanks
>
>
>
+
Hari Shreedharan 2012-08-23, 18:57
+
Bhaskar V. Karambelkar 2012-08-23, 22:28
+
Hari Shreedharan 2012-08-23, 22:45
+
Bhaskar V. Karambelkar 2012-08-23, 22:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB