Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: heterogenous kafka cluster?


Copy link to this message
-
Re: heterogenous kafka cluster?
Yeah,

I thought of that (running 2 kafkas on one box), but it doesn't really add
the benefit of redundancy through replication (e.g. if we have 2 replicas
mapping to the same physical machine).

Jason
On Fri, May 17, 2013 at 2:50 PM, Chris Riccomini <[EMAIL PROTECTED]>wrote:

> Hey guys,
>
> I have no idea if this would be reasonable, but what about just running
> two Kafka processes on the bigger box?
>
> Cheers,
> Chris
>
> On 5/17/13 2:48 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:
>
> >Just resource allocation issues.  E.g. imagine having an existing kafka
> >cluster with one machine spec, and getting access to a few more hosts to
> >augment the cluster, which are newer and therefore have twice the disk
> >storage.  I'd like to seamlessly add them into the cluster, without having
> >to replace everything en masse.  Thus, it would be nice for the newer ones
> >to take proportionally more load based on the relative storage available,
> >etc.
> >
> >Jason
> >
> >
> >On Fri, May 17, 2013 at 2:34 PM, Neha Narkhede
> ><[EMAIL PROTECTED]>wrote:
> >
> >> That does seem a little hacky. But I'm trying to understand the
> >>requirement
> >> behind having to deploy heterogeneous hardware. What are you trying to
> >> achieve or optimize?
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >>wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm wondering if there's a good way to have a heterogenous kafka
> >>cluster
> >> > (specifically, if we have nodes with different sized disks).  So, we
> >> might
> >> > want a larger node to receive more messages than a smaller node, etc.
> >> >
> >> > I expect there's something we can do with using a partitioner that has
> >> > specific knowledge about the hosts in the cluster, but this feels
> >>messy,
> >> to
> >> > have this config on every producer client....
> >> >
> >> > Thoughts?
> >> >
> >> > Jason
> >> >
> >>
>
>

 
+
Francis Dallaire 2013-05-17, 23:28
+
Jason Rosenberg 2013-05-17, 21:30
+
Neha Narkhede 2013-05-17, 21:35
+
Jason Rosenberg 2013-05-17, 21:48
+
Jun Rao 2013-05-18, 03:38
+
Jason Rosenberg 2013-05-18, 06:28
+
Neha Narkhede 2013-05-23, 21:31
+
Maxime Brugidou 2013-05-23, 21:37
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB