Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> changing broker hosts with 0.7.2


+
Jason Rosenberg 2013-03-19, 21:04
Copy link to this message
-
Re: changing broker hosts with 0.7.2
Can you do the following -

1. Start a mirror Kafka cluster with the new version on a separate
zookeeper namespace. Configure this to mirror data from the existing kafka
cluster.
2. Move your consumers to pull data from the mirror
3. For each producer, one at a time, change the zookeeper namespace to
point to the mirror and restart the producer.
4. Once the producers have moved to mirror cluster, shutdown mirroring and
old cluster.

Thanks,
Neha

On Tuesday, March 19, 2013, Jason Rosenberg wrote:

> I need to upgrade some kafka broker servers.  So I need to seamlessly
> migrate traffic from the old brokers to the new ones, without losing data,
> and without stopping producers.  I can temporarily stop consumers, etc.
>
> Is there a strategy for this?
>
> Also, because of the way we are embedding kafka in our framework, our
> brokerId's are auto-generated (based on hostname, etc.), so I can't simply
> copy over broker log files, etc., by transferring an old brokerId to a new
> host.
>
> Is there a way to change the view of the cluster from the producer's
> standpoint, without doing so from the consumers standpoint?  That way, the
> producers can start writing to the new brokers, while the consumers drain
> all data from the old brokers before switching to the new brokers.
>
> I don't actually care about ordering of messages, since the consumers are
> publishing them to a store that will index them properly based on source
> timestamp, etc.
>
> We are using zk for both producers and consumers connections.
>
> This is using 0.7.2.  I assume in 0.8 it will be easier, since with
> replication, you can phase in the new servers gradually, etc., no?
>
> Thanks,
>
> Jason
>

 
+
Jason Rosenberg 2013-03-20, 06:26
+
Neha Narkhede 2013-03-20, 13:42
+
Jason Rosenberg 2013-03-20, 15:56
+
Neha Narkhede 2013-03-20, 16:02
+
Philip OToole 2013-03-20, 16:07
+
Jason Rosenberg 2013-03-20, 17:55
+
Philip OToole 2013-03-20, 18:10
+
Jason Rosenberg 2013-03-20, 18:21
+
Philip OToole 2013-03-20, 19:00
+
Jason Rosenberg 2013-03-20, 19:07
+
Philip OToole 2013-03-20, 19:16
+
Jason Rosenberg 2013-03-20, 19:33
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB