Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: heterogenous kafka cluster?


+
Francis Dallaire 2013-05-17, 23:28
+
Jason Rosenberg 2013-05-17, 21:30
+
Neha Narkhede 2013-05-17, 21:35
+
Jason Rosenberg 2013-05-17, 21:48
+
Chris Riccomini 2013-05-17, 21:50
+
Jason Rosenberg 2013-05-17, 22:26
+
Jun Rao 2013-05-18, 03:38
Copy link to this message
-
Re: heterogenous kafka cluster?
Letting each broker have a weight sounds like a great idea.

Since in my use case, topics are generally auto-created, it won't be
practical to map brokers manually per topic.

Thanks,

Jason
On Fri, May 17, 2013 at 8:38 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> In 0.8, you can create topics manually and explicitly specify the replica
> to broker mapping. Post 0.8, we can think of some more automated ways to
> deal with this (e.g., let each broker carry some kind of weight).
>
> Thanks,
>
> Jun
>
>
> On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I'm wondering if there's a good way to have a heterogenous kafka cluster
> > (specifically, if we have nodes with different sized disks).  So, we
> might
> > want a larger node to receive more messages than a smaller node, etc.
> >
> > I expect there's something we can do with using a partitioner that has
> > specific knowledge about the hosts in the cluster, but this feels messy,
> to
> > have this config on every producer client....
> >
> > Thoughts?
> >
> > Jason
> >
>

 
+
Neha Narkhede 2013-05-23, 21:31
+
Maxime Brugidou 2013-05-23, 21:37