Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> heterogenous kafka cluster?

Copy link to this message
Re: heterogenous kafka cluster?
Hey guys,

I have no idea if this would be reasonable, but what about just running
two Kafka processes on the bigger box?


On 5/17/13 2:48 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:

>Just resource allocation issues.  E.g. imagine having an existing kafka
>cluster with one machine spec, and getting access to a few more hosts to
>augment the cluster, which are newer and therefore have twice the disk
>storage.  I'd like to seamlessly add them into the cluster, without having
>to replace everything en masse.  Thus, it would be nice for the newer ones
>to take proportionally more load based on the relative storage available,
>On Fri, May 17, 2013 at 2:34 PM, Neha Narkhede
>> That does seem a little hacky. But I'm trying to understand the
>> behind having to deploy heterogeneous hardware. What are you trying to
>> achieve or optimize?
>> Thanks,
>> Neha
>> On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> > Hi,
>> >
>> > I'm wondering if there's a good way to have a heterogenous kafka
>> > (specifically, if we have nodes with different sized disks).  So, we
>> might
>> > want a larger node to receive more messages than a smaller node, etc.
>> >
>> > I expect there's something we can do with using a partitioner that has
>> > specific knowledge about the hosts in the cluster, but this feels
>> to
>> > have this config on every producer client....
>> >
>> > Thoughts?
>> >
>> > Jason
>> >