Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Can't connect to a server if not enough partitions


+
Jason Rosenberg 2013-05-09, 05:16
+
Neha Narkhede 2013-05-09, 05:57
Copy link to this message
-
Re: Can't connect to a server if not enough partitions
Jason Rosenberg 2013-05-09, 06:51
Neha,

Thanks, I think I did understand what was going (despite the error
message).  And my question stands, if a broker is momentarily down,
shouldn't we still be able to create a topic?  If we send a message to a
topic it will succeed, even if not all replicas are available.  Why should
the initial message be any different?

Jason
On Wed, May 8, 2013 at 10:57 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> I think this error message is somewhat misleading since we create topic on
> the first metadata request. It is complaining that a topic with the
> required replication factor cannot be created if there aren't enough
> brokers to satisfy the replication factor. This is expected behavior
> whether you use auto creation of topics or manual creation. However, the
> metadata requests will always give you correct information about existing
> topics.
>
> Thanks,
> Neha
>
>
> On Wed, May 8, 2013 at 10:15 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > With 0.8.0, I'm seeing that an initial metadata request fails, if the
> > number of running brokers is fewer than the configured replication
> factor:
> >
> > 877 [kafka-request-handler-0] ERROR kafka.server.KafkaApis  -
> > [KafkaApi-1946108683] Error while retrieving topic metadata
> > kafka.admin.AdministrationException: replication factor: 2 larger than
> > available brokers: 1
> > at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:62)
> > at
> kafka.admin.CreateTopicCommand$.createTopic(CreateTopicCommand.scala:92)
> > at
> >
> >
> kafka.server.KafkaApis$$anonfun$handleTopicMetadataRequest$1.apply(KafkaApis.scala:409)
> > at
> >
> >
> kafka.server.KafkaApis$$anonfun$handleTopicMetadataRequest$1.apply(KafkaApis.scala:401)
> > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:400)
> > at kafka.server.KafkaApis.handle(KafkaApis.scala:61)
> > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:41)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > However, if after connecting, the number of brokers goes down, producing
> > clients have no problems continuing sending messages, etc.
> >
> > So, I thought the idea was that once a replica becomes available, it will
> > be caught up with messages it might have missed, etc.  This is good
> because
> > it makes doing things like rolling restarts of the brokers possible, etc.
> >  But it's a problem if a rolling restart happens at the same time a new
> > client is coming online to try and initialize a connection.
> >
> > Thoughts?
> >
> > Shouldn't the requirements be the same for initial connections as ongoing
> > connections?
> >
> > Jason
> >
>

 
+
Jun Rao 2013-05-09, 15:00