Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka and Zookeeper node removal from two nodes Kafka cluster


Copy link to this message
-
Re: Kafka and Zookeeper node removal from two nodes Kafka cluster
When creating a new topic, we require # live brokers to be equal to or
larger than # replicas. Without enough brokers, can't complete the replica
assignment since we can't assign more than 1 replica on the same broker.

Thanks,

Jun
On Tue, Oct 15, 2013 at 1:47 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Is there a fundamental reason for not allowing creation of new topics while
> in an under-replicated state?  For systems that use automatic topic
> creation, it seems like losing a node in this case is akin to the cluster
> being unavailable, if one of the nodes goes down, etc.
>
>
> On Tue, Oct 15, 2013 at 1:25 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > Steve - that's right. I think Monika wanted clarification on what
> > would happen if replication factor is two and only one broker is
> > available. In that case, you won't be able to create new topics with
> > replication factor two (you should see an AdministrationException
> > saying the replication factor is larger than available brokers).
> >
> > However, you can send messages to available partitions of topics that
> > have already been created - because the ISR would shrink to only one
> > replica for those topics - although the cluster would be in an
> > under-replicated state. This is covered in the documentation
> > (http://kafka.apache.org/documentation.html#replication) under the
> > discussion about ISR.
> >
> > Thanks,
> >
> > Joel
> >
> > On Tue, Oct 15, 2013 at 10:19 AM, Steve Morin <[EMAIL PROTECTED]>
> > wrote:
> > > If you have a double broker failure with replication factor of 2 and
> only
> > > have 2 brokers in the cluster.  Wouldn't every partition be not
> > available?
> > >
> > >
> > > On Tue, Oct 15, 2013 at 8:48 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > >> If you have double broker failures with a replication factor of 2,
> some
> > >> partitions will not be available. When one of the brokers comes back,
> > the
> > >> partition is made available again (there is potential data loss), but
> > in an
> > >> under replicated mode. After the second broker comes back, it will
> > catch up
> > >> from the other replica and the partition will eventually be fully
> > >> replicated. There is no need to change the replication factor during
> > this
> > >> process.
> > >>
> > >> As for ZK, you can always use the full connection string. ZK will pick
> > live
> > >> servers to establish connections.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Tue, Oct 15, 2013 at 3:46 AM, Monika Garg <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >> > I have 2 nodes kafka cluster with default.replication.factor=2,is
> set
> > in
> > >> > server.properties file.
> > >> >
> > >> > I removed one node-in removing that node,I killed Kafka
> > process,removed
> > >> all
> > >> > the kafka-logs and bundle from that node.
> > >> >
> > >> > Then I stopped my remaining running node in the cluster and started
> > >> > again(default.replication.factor is still set to 2 in this node
> > >> > server.properties file).
> > >> > I was expecting some error/exception as now I don't have two nodes
> in
> > my
> > >> > cluster.But I didn't get any error/exception and my node
> successfully
> > >> > started and I am able to create topics on it.
> > >> >
> > >> > So should the "default.replication.factor" be updated from
> > >> > "default.replication.factor=2" to "default.replication.factor=1" ,
> in
> > the
> > >> > remaining running node?
> > >> >
> > >> > Similarly if there are two external zookeeper
> > >> > nodes(zookeeper.connect=host1:port1,host2:port1) in my cluster and
> > now I
> > >> > have removed one zookeeper node(host1:port1) from the cluster,So
> > should
> > >> the
> > >> > property "zookeeper.connect" be updated from
> > >> > (zookeeper.connect=host1:port1,host2:port1) to
> > >> > (zookeeper.connect=host2:port1)?
> > >> >
> > >> > --
> > >> > *Moniii*
> > >> >
> > >>
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB