Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - changing broker hosts with 0.7.2


Copy link to this message
-
Re: changing broker hosts with 0.7.2
Neha Narkhede 2013-03-20, 13:42
The zookeeper connection URL with namespace can be
zkhost1:123,zkhost2:123/newnamespace

The wiki is up to date for Kafka 0.7.2. There is no officially supported
feature to do that sort of migration, I suggested one approach that I could
think of :-)

Thanks,
Neha

On Tuesday, March 19, 2013, Jason Rosenberg wrote:

> I can do most of that I presume.
>
> It looks like to set up a separate namespace for zk, I can add /path at the
> end of each node:port in my zkconnect string, e.g.:
>  zkhost1:123/newnamespace,zkhost2:123/newnamespace
> right?
>
> For mirroring, there's some vague documentation here:
> https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
> Is this the most up to date approach for 0.7.2?  Set up a MirrorMaker
> intermediate process that consumes from the old and produces to the new?
>
> I am not able to restart producers one by one (as there are many, on a
> rather asynchronous update/restart cycle).  But I can eventually get them
> migrated over, etc.
>
> Jason
>
> On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <[EMAIL PROTECTED]<javascript:;>
> >wrote:
>
> > Can you do the following -
> >
> > 1. Start a mirror Kafka cluster with the new version on a separate
> > zookeeper namespace. Configure this to mirror data from the existing
> kafka
> > cluster.
> > 2. Move your consumers to pull data from the mirror
> > 3. For each producer, one at a time, change the zookeeper namespace to
> > point to the mirror and restart the producer.
> > 4. Once the producers have moved to mirror cluster, shutdown mirroring
> and
> > old cluster.
> >
> > Thanks,
> > Neha
> >
> > On Tuesday, March 19, 2013, Jason Rosenberg wrote:
> >
> > > I need to upgrade some kafka broker servers.  So I need to seamlessly
> > > migrate traffic from the old brokers to the new ones, without losing
> > data,
> > > and without stopping producers.  I can temporarily stop consumers, etc.
> > >
> > > Is there a strategy for this?
> > >
> > > Also, because of the way we are embedding kafka in our framework, our
> > > brokerId's are auto-generated (based on hostname, etc.), so I can't
> > simply
> > > copy over broker log files, etc., by transferring an old brokerId to a
> > new
> > > host.
> > >
> > > Is there a way to change the view of the cluster from the producer's
> > > standpoint, without doing so from the consumers standpoint?  That way,
> > the
> > > producers can start writing to the new brokers, while the consumers
> drain
> > > all data from the old brokers before switching to the new brokers.
> > >
> > > I don't actually care about ordering of messages, since the consumers
> are
> > > publishing them to a store that will index them properly based on
> source
> > > timestamp, etc.
> > >
> > > We are using zk for both producers and consumers connections.
> > >
> > > This is using 0.7.2.  I assume in 0.8 it will be easier, since with
> > > replication, you can phase in the new servers gradually, etc., no?
> > >
> > > Thanks,
> > >
> > > Jason
> > >
> >
>