Neha Narkhede 2013-03-20, 02:07
-Re: changing broker hosts with 0.7.2
Jason Rosenberg 2013-03-20, 06:26
I can do most of that I presume.
It looks like to set up a separate namespace for zk, I can add /path at the
end of each node:port in my zkconnect string, e.g.:
For mirroring, there's some vague documentation here:
Is this the most up to date approach for 0.7.2? Set up a MirrorMaker
intermediate process that consumes from the old and produces to the new?
I am not able to restart producers one by one (as there are many, on a
rather asynchronous update/restart cycle). But I can eventually get them
migrated over, etc.
On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
> Can you do the following -
> 1. Start a mirror Kafka cluster with the new version on a separate
> zookeeper namespace. Configure this to mirror data from the existing kafka
> 2. Move your consumers to pull data from the mirror
> 3. For each producer, one at a time, change the zookeeper namespace to
> point to the mirror and restart the producer.
> 4. Once the producers have moved to mirror cluster, shutdown mirroring and
> old cluster.
> On Tuesday, March 19, 2013, Jason Rosenberg wrote:
> > I need to upgrade some kafka broker servers. So I need to seamlessly
> > migrate traffic from the old brokers to the new ones, without losing
> > and without stopping producers. I can temporarily stop consumers, etc.
> > Is there a strategy for this?
> > Also, because of the way we are embedding kafka in our framework, our
> > brokerId's are auto-generated (based on hostname, etc.), so I can't
> > copy over broker log files, etc., by transferring an old brokerId to a
> > host.
> > Is there a way to change the view of the cluster from the producer's
> > standpoint, without doing so from the consumers standpoint? That way,
> > producers can start writing to the new brokers, while the consumers drain
> > all data from the old brokers before switching to the new brokers.
> > I don't actually care about ordering of messages, since the consumers are
> > publishing them to a store that will index them properly based on source
> > timestamp, etc.
> > We are using zk for both producers and consumers connections.
> > This is using 0.7.2. I assume in 0.8 it will be easier, since with
> > replication, you can phase in the new servers gradually, etc., no?
> > Thanks,
> > Jason
Neha Narkhede 2013-03-20, 13:42
Jason Rosenberg 2013-03-20, 15:56
Neha Narkhede 2013-03-20, 16:02
Philip O'Toole 2013-03-20, 16:07
Jason Rosenberg 2013-03-20, 17:55
Philip O'Toole 2013-03-20, 18:10
Jason Rosenberg 2013-03-20, 18:21
Philip O'Toole 2013-03-20, 19:00
Jason Rosenberg 2013-03-20, 19:07
Philip O'Toole 2013-03-20, 19:16
Jason Rosenberg 2013-03-20, 19:33