Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: heterogenous kafka cluster?


+
Jason Rosenberg 2013-05-17, 22:26
+
Francis Dallaire 2013-05-17, 23:28
+
Jason Rosenberg 2013-05-17, 21:30
+
Neha Narkhede 2013-05-17, 21:35
+
Jason Rosenberg 2013-05-17, 21:48
+
Jun Rao 2013-05-18, 03:38
+
Jason Rosenberg 2013-05-18, 06:28
+
Neha Narkhede 2013-05-23, 21:31
Copy link to this message
-
Re: heterogenous kafka cluster?
Have you thought about integrating Kafka into a distributed resource
management framework like Hadoop YARN (which would probably leverage HDFS)
or Mesos?
On May 23, 2013 11:31 PM, "Neha Narkhede" <[EMAIL PROTECTED]> wrote:

> This paper talks about how to do that -
> http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf
> It will be interesting to see what part of it Kafka can adopt, if any.
>
> Thanks,
> Neha
>
>
> On Fri, May 17, 2013 at 11:28 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > Letting each broker have a weight sounds like a great idea.
> >
> > Since in my use case, topics are generally auto-created, it won't be
> > practical to map brokers manually per topic.
> >
> > Thanks,
> >
> > Jason
> >
> >
> > On Fri, May 17, 2013 at 8:38 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > In 0.8, you can create topics manually and explicitly specify the
> replica
> > > to broker mapping. Post 0.8, we can think of some more automated ways
> to
> > > deal with this (e.g., let each broker carry some kind of weight).
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm wondering if there's a good way to have a heterogenous kafka
> > cluster
> > > > (specifically, if we have nodes with different sized disks).  So, we
> > > might
> > > > want a larger node to receive more messages than a smaller node, etc.
> > > >
> > > > I expect there's something we can do with using a partitioner that
> has
> > > > specific knowledge about the hosts in the cluster, but this feels
> > messy,
> > > to
> > > > have this config on every producer client....
> > > >
> > > > Thoughts?
> > > >
> > > > Jason
> > > >
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB