Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> changing broker hosts with 0.7.2


Copy link to this message
-
Re: changing broker hosts with 0.7.2
On Wed, Mar 20, 2013 at 9:06 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> On Wed, Mar 20, 2013 at 8:41 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> > I think might be cool, would be to have a feature where by you can tell a
> > broker to stop accepting new data produced to it, but still allow
> consumers
> > to consume from it.
> >
> > That way, you can roll out new brokers to a cluster, turn off producing
> to
> > the old nodes, then wait for the log retention period, and then remove
> the
> > old nodes from the cluster.
> >
> > Does that make sense?  Could it be easily done?
>
> Does to me. A way to do this is to place a load-balancer between your
> consumers and brokers, allowing individual brokers to be taken out of
> rotation for maintenance.
Not sure I understand.  I need for a node to be taken out of the pool that
producers produce to, but still need consumers consuming from all brokers,
while we drain data from the brokers to be replaced.

Since I am using zk for producers to discover brokers, there's not an easy
way to tell producers to stop producing to a sub-set of nodes, without also
having the same affect on consumers.

Maybe I should first switch all producers to use a brokerlist, pointing to
only the new hosts.  But still have all the hosts in zk.....that might
work, I should think.  But then I'm making the commitment not to be using
zk to connect to the brokers for producers.

Jason