Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Partition reassignment stucked


Copy link to this message
-
Re: Partition reassignment stucked
Neha Narkhede 2013-12-09, 21:21
We will announce it on this mailing list. It is probably a month away from
a release.

Thanks,
Neha
On Mon, Dec 9, 2013 at 12:02 PM, Maxime Nay <[EMAIL PROTECTED]> wrote:

> Ok, thanks for your help. When 0.8.1 will be production ready, will you
> announce it somewhere (will you release it right away) ?
>
> Thanks,
> Maxime
>
>
>
> 2013/12/9 Neha Narkhede <[EMAIL PROTECTED]>
>
> > I wouldn't call 0.8.1 production ready just yet. We are still in the
> > process of deploying it at LinkedIn. Until it is ready, there isn't a
> good
> > cluster expansion solution other than spinning up a new cluster. This is
> > probably a little easier if you have a VIP in front of your kafka
> cluster.
> >
> > Thanks,
> > Neha
> >
> >
> > On Mon, Dec 9, 2013 at 9:40 AM, Maxime Nay <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > We used the code checked in this branch a few hours before the official
> > > 0.8.0 final release : https://github.com/apache/kafka/tree/0.8
> > > So hopefully it should be the exact same code as the official release.
> > >
> > > The controller logs are empty.
> > >
> > > In a previous exchange you advised us to not use trunk unless we felt
> > > adventurous.
> > > So if you think 0.8.1 is production ready, and that it shouldn't crash
> > > unexpectedly, we can probably give it a try. Otherwise, since our data
> > > pipeline heavily relies on Kafka, we might have to use another
> > solution...
> > > (bring up a brand new cluster each time we want to resize the cluster
> ?)
> > >
> > > Thanks
> > > Maxime
> > >
> > >
> > >
> > > 2013/12/8 Neha Narkhede <[EMAIL PROTECTED]>
> > >
> > > > Unfortunately quite a few bugs with reassignment are fixed only in
> > > 0.8.1. I
> > > > wonder if you can run trunk and see how that goes?
> > > >
> > > > Thanks,
> > > > Neha
> > > > On Dec 6, 2013 9:46 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Are you using the 0.8.0 final release? Any error in controller log?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Fri, Dec 6, 2013 at 4:38 PM, Maxime Nay <[EMAIL PROTECTED]>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are trying to add a broker to a 10 node cluster. We have 7
> > > different
> > > > > > topics, each of them is divided in 10 partitions, and their
> > > replication
> > > > > > factor is 3.
> > > > > >
> > > > > > To send traffic to this new node, we tried the
> > > > > kafka-reassign-partitions.sh
> > > > > > tool, but for some reason, it doesnt work, and now it seems that
> we
> > > are
> > > > > > stuck in some kind of partition reassignment process.
> > > > > > We can't see any data on our new node, and when we try to execute
> > the
> > > > > > partition reassignment tool for another topic it says :
> > > > > >
> > > > > > Partitions reassignment failed due to Partition reassignment
> > > currently
> > > > in
> > > > > > progress for Map([AD_EVENTS,5] ->
> > > > > > ReassignedPartitionsContext(List(2),null), [AD_EVENTS,3] ->
> > > > > > ReassignedPartitionsContext(List(1),null), [AD_EVENTS,4] ->
> > > > > > ReassignedPartitionsContext(List(3),null)). Aborting operation
> > > > > > kafka.common.AdminCommandFailedException: Partition reassignment
> > > > > currently
> > > > > > in progress for Map([AD_EVENTS,5] ->
> > > > > > ReassignedPartitionsContext(List(2),null), [AD_EVENTS,3] ->
> > > > > > ReassignedPartitionsContext(List(1),null), [AD_EVENTS,4] ->
> > > > > > ReassignedPartitionsContext(List(3),null)). Aborting operation
> > > > > >     at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:201)
> > > > > >     at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:140)
> > > > > >     at
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
> >