Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - changing broker hosts with 0.7.2


+
Jason Rosenberg 2013-03-19, 21:04
+
Neha Narkhede 2013-03-20, 02:07
+
Jason Rosenberg 2013-03-20, 06:26
+
Neha Narkhede 2013-03-20, 13:42
+
Jason Rosenberg 2013-03-20, 15:56
+
Neha Narkhede 2013-03-20, 16:02
+
Philip OToole 2013-03-20, 16:07
+
Jason Rosenberg 2013-03-20, 17:55
+
Philip OToole 2013-03-20, 18:10
Copy link to this message
-
Re: changing broker hosts with 0.7.2
Jason Rosenberg 2013-03-20, 18:21
On Wed, Mar 20, 2013 at 11:10 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> On Wed, Mar 20, 2013 at 10:55 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > On Wed, Mar 20, 2013 at 9:06 AM, Philip O'Toole <[EMAIL PROTECTED]>
> wrote:
> >
> >> On Wed, Mar 20, 2013 at 8:41 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >> > I think might be cool, would be to have a feature where by you can
> tell a
> >> > broker to stop accepting new data produced to it, but still allow
> >> consumers
> >> > to consume from it.
> >> >
> >> > That way, you can roll out new brokers to a cluster, turn off
> producing
> >> to
> >> > the old nodes, then wait for the log retention period, and then remove
> >> the
> >> > old nodes from the cluster.
> >> >
> >> > Does that make sense?  Could it be easily done?
> >>
> >> Does to me. A way to do this is to place a load-balancer between your
> >> consumers and brokers, allowing individual brokers to be taken out of
> >> rotation for maintenance.
> >
> >
> > Not sure I understand.  I need for a node to be taken out of the pool
> that
> > producers produce to, but still need consumers consuming from all
> brokers,
> > while we drain data from the brokers to be replaced.
>
> Taking it out from behind the load-balancer between Producers and
> Kafka means that Producers can no longer write to it. I said nothing
> about disconnecting the *Consumers*. :-)
>
>
No worries Philip, I'll assume you you mispoke at first when talking about
a load-balancer between the consumers and brokers.  Kafka, unfortunately,
doesn't allow consumers to connect to kafka via a load balancer.  For
producers, also, you can't really use a load-balancer to connect to brokers
(you can use zk, or you can use a broker list, in 0.7.2, and in 0.8, you
can use an LB for the initial meta data connection, but then you still have
to have direct connections to each broker from each producer).

Jason

 
+
Philip OToole 2013-03-20, 19:00
+
Jason Rosenberg 2013-03-20, 19:07
+
Philip OToole 2013-03-20, 19:16
+
Jason Rosenberg 2013-03-20, 19:33