Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> heterogenous kafka cluster?


Copy link to this message
-
Re: heterogenous kafka cluster?
Have you thought about integrating Kafka into a distributed resource
management framework like Hadoop YARN (which would probably leverage HDFS)
or Mesos?
On May 23, 2013 11:31 PM, "Neha Narkhede" <[EMAIL PROTECTED]> wrote:

> This paper talks about how to do that -
> http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf
> It will be interesting to see what part of it Kafka can adopt, if any.
>
> Thanks,
> Neha
>
>
> On Fri, May 17, 2013 at 11:28 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > Letting each broker have a weight sounds like a great idea.
> >
> > Since in my use case, topics are generally auto-created, it won't be
> > practical to map brokers manually per topic.
> >
> > Thanks,
> >
> > Jason
> >
> >
> > On Fri, May 17, 2013 at 8:38 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > In 0.8, you can create topics manually and explicitly specify the
> replica
> > > to broker mapping. Post 0.8, we can think of some more automated ways
> to
> > > deal with this (e.g., let each broker carry some kind of weight).
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm wondering if there's a good way to have a heterogenous kafka
> > cluster
> > > > (specifically, if we have nodes with different sized disks).  So, we
> > > might
> > > > want a larger node to receive more messages than a smaller node, etc.
> > > >
> > > > I expect there's something we can do with using a partitioner that
> has
> > > > specific knowledge about the hosts in the cluster, but this feels
> > messy,
> > > to
> > > > have this config on every producer client....
> > > >
> > > > Thoughts?
> > > >
> > > > Jason
> > > >
> > >
> >
>