Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - heterogenous kafka cluster?


Copy link to this message
-
Re: heterogenous kafka cluster?
Chris Riccomini 2013-05-17, 21:50
Hey guys,

I have no idea if this would be reasonable, but what about just running
two Kafka processes on the bigger box?

Cheers,
Chris

On 5/17/13 2:48 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:

>Just resource allocation issues.  E.g. imagine having an existing kafka
>cluster with one machine spec, and getting access to a few more hosts to
>augment the cluster, which are newer and therefore have twice the disk
>storage.  I'd like to seamlessly add them into the cluster, without having
>to replace everything en masse.  Thus, it would be nice for the newer ones
>to take proportionally more load based on the relative storage available,
>etc.
>
>Jason
>
>
>On Fri, May 17, 2013 at 2:34 PM, Neha Narkhede
><[EMAIL PROTECTED]>wrote:
>
>> That does seem a little hacky. But I'm trying to understand the
>>requirement
>> behind having to deploy heterogeneous hardware. What are you trying to
>> achieve or optimize?
>>
>> Thanks,
>> Neha
>>
>>
>> On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>>wrote:
>>
>> > Hi,
>> >
>> > I'm wondering if there's a good way to have a heterogenous kafka
>>cluster
>> > (specifically, if we have nodes with different sized disks).  So, we
>> might
>> > want a larger node to receive more messages than a smaller node, etc.
>> >
>> > I expect there's something we can do with using a partitioner that has
>> > specific knowledge about the hosts in the cluster, but this feels
>>messy,
>> to
>> > have this config on every producer client....
>> >
>> > Thoughts?
>> >
>> > Jason
>> >
>>