Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> heterogenous kafka cluster?


Copy link to this message
-
Re: heterogenous kafka cluster?
Yeah,

I thought of that (running 2 kafkas on one box), but it doesn't really add
the benefit of redundancy through replication (e.g. if we have 2 replicas
mapping to the same physical machine).

Jason
On Fri, May 17, 2013 at 2:50 PM, Chris Riccomini <[EMAIL PROTECTED]>wrote:

> Hey guys,
>
> I have no idea if this would be reasonable, but what about just running
> two Kafka processes on the bigger box?
>
> Cheers,
> Chris
>
> On 5/17/13 2:48 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:
>
> >Just resource allocation issues.  E.g. imagine having an existing kafka
> >cluster with one machine spec, and getting access to a few more hosts to
> >augment the cluster, which are newer and therefore have twice the disk
> >storage.  I'd like to seamlessly add them into the cluster, without having
> >to replace everything en masse.  Thus, it would be nice for the newer ones
> >to take proportionally more load based on the relative storage available,
> >etc.
> >
> >Jason
> >
> >
> >On Fri, May 17, 2013 at 2:34 PM, Neha Narkhede
> ><[EMAIL PROTECTED]>wrote:
> >
> >> That does seem a little hacky. But I'm trying to understand the
> >>requirement
> >> behind having to deploy heterogeneous hardware. What are you trying to
> >> achieve or optimize?
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >>wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm wondering if there's a good way to have a heterogenous kafka
> >>cluster
> >> > (specifically, if we have nodes with different sized disks).  So, we
> >> might
> >> > want a larger node to receive more messages than a smaller node, etc.
> >> >
> >> > I expect there's something we can do with using a partitioner that has
> >> > specific knowledge about the hosts in the cluster, but this feels
> >>messy,
> >> to
> >> > have this config on every producer client....
> >> >
> >> > Thoughts?
> >> >
> >> > Jason
> >> >
> >>
>
>