Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Failure scenarios and consequences


Copy link to this message
-
Re: Failure scenarios and consequences
Hi Jeremy -

One note in-line:

On 9 December 2010 12:04, Mahadev Konar <[EMAIL PROTECTED]> wrote:

> Hi Jeremy,
>   Responses in line below:
>
> On 12/9/10 11:53 AM, "Jeremy Hanna" <[EMAIL PROTECTED]> wrote:
>
> I looked around on the wiki and in the user list archives and couldn't find
> something definitive about certain failure scenarios.
>
> A partition splits the ensemble where a quorum is on one side of the
> partition
> -- if the leader is on the quorum side of the partition, what happens to
> reads/writes that go to the non-quorum side?  I assume writes return errors
> because it can't get to the leader.  Reads?
>
> > The reads will also fail on all the quorum nodes until a new quorum is
> elected.
>
>
This is true, but since reads are served locally and are not serialised by
the leader I believe there is a small time window during which a network
partition may have occurred and a follower may not have realised it, so the
follower keeps on serving reads for slightly longer than it would serve
writes for. In most cases the time of failure detection is very short, so
this wouldn't be obvious, but if you turned down the ping frequency from
followers to the leader then you could engineer an arbitrarily large gap
when reads would be served. Note that no consistency guarantees are violated
here because it's legal to serve a stale value as long as you yourself
haven't overwritten it. Overwriting it would trigger a failure detection and
no subsequent reads would be served.

Writes are guaranteed not to get through on the smaller side of the
partition, because every write must be acknowledged by a quorum of nodes
before it is committed. In the case of a network partition, this is
obviously not possible on the smaller side.

Henry

> -- if the leader is on the non-quorum side of the partition, I would assume
> that the quorum side of the partition would elect a new leader for those
> clients on its side of the partition.  However, is there the possibility for
> the leader on the non-quorum side to accept writes before it realizes that
> there's no longer a quorum?  Just wondering about the possibility of
> corruption and then when the cluster syncs back up how the cluster would
> handle that data.
>
> > No there isnt. The leader relinquishes its right as a leader as soon as
> it realizes a quorum isnt committing the changes it proposed.
>
> (I would be happy to create a wiki page for failure scenarios if one
> doesn't exist that people could add to, but maybe this is just common
> knowledge.)
>
> > Please do!
>
> thanks
> mahadev
>

--
Henry Robinson
Software Engineer
Cloudera
415-994-6679