Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: heterogenous kafka cluster?


+
Jason Rosenberg 2013-05-17, 22:26
+
Francis Dallaire 2013-05-17, 23:28
+
Jason Rosenberg 2013-05-17, 21:30
+
Neha Narkhede 2013-05-17, 21:35
+
Jason Rosenberg 2013-05-17, 21:48
+
Jun Rao 2013-05-18, 03:38
+
Jason Rosenberg 2013-05-18, 06:28
Copy link to this message
-
Re: heterogenous kafka cluster?
This paper talks about how to do that -
http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf
It will be interesting to see what part of it Kafka can adopt, if any.

Thanks,
Neha
On Fri, May 17, 2013 at 11:28 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Letting each broker have a weight sounds like a great idea.
>
> Since in my use case, topics are generally auto-created, it won't be
> practical to map brokers manually per topic.
>
> Thanks,
>
> Jason
>
>
> On Fri, May 17, 2013 at 8:38 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > In 0.8, you can create topics manually and explicitly specify the replica
> > to broker mapping. Post 0.8, we can think of some more automated ways to
> > deal with this (e.g., let each broker carry some kind of weight).
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi,
> > >
> > > I'm wondering if there's a good way to have a heterogenous kafka
> cluster
> > > (specifically, if we have nodes with different sized disks).  So, we
> > might
> > > want a larger node to receive more messages than a smaller node, etc.
> > >
> > > I expect there's something we can do with using a partitioner that has
> > > specific knowledge about the hosts in the cluster, but this feels
> messy,
> > to
> > > have this config on every producer client....
> > >
> > > Thoughts?
> > >
> > > Jason
> > >
> >
>

 
+
Maxime Brugidou 2013-05-23, 21:37