Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> changing broker hosts with 0.7.2


Copy link to this message
-
changing broker hosts with 0.7.2
I need to upgrade some kafka broker servers.  So I need to seamlessly
migrate traffic from the old brokers to the new ones, without losing data,
and without stopping producers.  I can temporarily stop consumers, etc.

Is there a strategy for this?

Also, because of the way we are embedding kafka in our framework, our
brokerId's are auto-generated (based on hostname, etc.), so I can't simply
copy over broker log files, etc., by transferring an old brokerId to a new
host.

Is there a way to change the view of the cluster from the producer's
standpoint, without doing so from the consumers standpoint?  That way, the
producers can start writing to the new brokers, while the consumers drain
all data from the old brokers before switching to the new brokers.

I don't actually care about ordering of messages, since the consumers are
publishing them to a store that will index them properly based on source
timestamp, etc.

We are using zk for both producers and consumers connections.

This is using 0.7.2.  I assume in 0.8 it will be easier, since with
replication, you can phase in the new servers gradually, etc., no?

Thanks,

Jason