Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - changing broker hosts with 0.7.2


Copy link to this message
-
Re: changing broker hosts with 0.7.2
Jason Rosenberg 2013-03-20, 06:26
I can do most of that I presume.

It looks like to set up a separate namespace for zk, I can add /path at the
end of each node:port in my zkconnect string, e.g.:
 zkhost1:123/newnamespace,zkhost2:123/newnamespace
right?

For mirroring, there's some vague documentation here:
https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
Is this the most up to date approach for 0.7.2?  Set up a MirrorMaker
intermediate process that consumes from the old and produces to the new?

I am not able to restart producers one by one (as there are many, on a
rather asynchronous update/restart cycle).  But I can eventually get them
migrated over, etc.

Jason

On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Can you do the following -
>
> 1. Start a mirror Kafka cluster with the new version on a separate
> zookeeper namespace. Configure this to mirror data from the existing kafka
> cluster.
> 2. Move your consumers to pull data from the mirror
> 3. For each producer, one at a time, change the zookeeper namespace to
> point to the mirror and restart the producer.
> 4. Once the producers have moved to mirror cluster, shutdown mirroring and
> old cluster.
>
> Thanks,
> Neha
>
> On Tuesday, March 19, 2013, Jason Rosenberg wrote:
>
> > I need to upgrade some kafka broker servers.  So I need to seamlessly
> > migrate traffic from the old brokers to the new ones, without losing
> data,
> > and without stopping producers.  I can temporarily stop consumers, etc.
> >
> > Is there a strategy for this?
> >
> > Also, because of the way we are embedding kafka in our framework, our
> > brokerId's are auto-generated (based on hostname, etc.), so I can't
> simply
> > copy over broker log files, etc., by transferring an old brokerId to a
> new
> > host.
> >
> > Is there a way to change the view of the cluster from the producer's
> > standpoint, without doing so from the consumers standpoint?  That way,
> the
> > producers can start writing to the new brokers, while the consumers drain
> > all data from the old brokers before switching to the new brokers.
> >
> > I don't actually care about ordering of messages, since the consumers are
> > publishing them to a store that will index them properly based on source
> > timestamp, etc.
> >
> > We are using zk for both producers and consumers connections.
> >
> > This is using 0.7.2.  I assume in 0.8 it will be easier, since with
> > replication, you can phase in the new servers gradually, etc., no?
> >
> > Thanks,
> >
> > Jason
> >
>