Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # user - Node being there and not at the same time


Copy link to this message
-
Re: Node being there and not at the same time
Alexander Shraer 2012-08-31, 05:21
Bill,

I'm sorry - you were right and I totally quoted the wrong place in the
code. The code that ensures that a client doesn't "go back in time" by
connecting to a server that is less up to date than that client is most
probably this one from ZooKeeperServer.java. I realized it after looking on
the question of Simon today in the mailing list...

     if (connReq.getLastZxidSeen() > zkDb.dataTree.lastProcessedZxid)

            String msg = "Refusing session request for client "

                + cnxn.getRemoteSocketAddress()

                + " as it has seen zxid 0x"

                + Long.toHexString(connReq.getLastZxidSeen())

                + " our last zxid is 0x"

                +
Long.toHexString(getZKDatabase().getDataTreeLastProcessedZxid())

                + " client must try another server";

On Mon, Aug 27, 2012 at 10:22 AM, Bill Bridge <[EMAIL PROTECTED]>wrote:

> Alex,
> You certainly know the code much better than I, so I may be mistaken here.
> It looks to me like waitForEpochAck() is about changes in the set of peers,
> and is not related to client connect/disconnects. I do not see how this
> would be called if a client disconnected due to some problem of his own,
> such as too slow to heartbeat, then reconnected to a different peer or
> observer.
>
> You suggest that a reconnecting client should ensure the new server has
> seen all transactions that the client has seen. This sounds like the right
> thing to do. This would certainly eliminate the race condition I
> postulated. This sounds like the kind of thing someone would have already
> thought of. If this is not already done then it would be a good change to
> make. I do not know where the code to do that would be. It could be part of
> the server reconnect code or it could be a sync() in the client library.
>
> If Mattias's code creates a new session when reconnecting, rather than
> reconnecting to the same session, then he could have the problem described
> even if reconnect ensures the client is not ahead of the server. He could
> fix this either by reconnecting to the same session, or simply doing a
> sync() when necessary.
>
> Thanks,
> Bill
>
>
> On 8/24/2012 6:11 PM, Alexander Shraer wrote:
>
>> Bill,  if I understand correctly this shouldn't be possible - the
>> client will not be able to connect to a server that is
>> less up-to-date than that same client. So if the create completed at
>> the client before it disconnects the new server will have to know
>> about it too otherwise the connection will fail. See
>> Leader.waitForEpochAck:
>>
>> if (ss.isMoreRecentThan(**leaderStateSummary)) {
>>                      throw new IOException("Follower is ahead of the
>> leader, leader summary: "
>>                                                      +
>> leaderStateSummary.**getCurrentEpoch()
>>                                                      + " (current epoch),
>> "
>>                                                      +
>> leaderStateSummary.**getLastZxid()
>>                                                      + " (last zxid)");
>>                  }
>>
>> of course its possible that another client connected to a different
>> server doesn't see the create.
>>
>> Alex
>>
>>
>> On Fri, Aug 24, 2012 at 5:15 PM, Bill Bridge <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Mattias,
>>>
>>> Is it possible that after you get NODEEXISTS from creation and before
>>> you do
>>> the second getData(), you reconnect to another ZooKeeper instance? If so,
>>> maybe the new connection is to a follower that has not yet seen the
>>> creation. If this is what is happening, then a sync() after the second
>>> NONODE with a third getData() should work. By only doing the sync() when
>>> you
>>> hit the unusual race condition it will have no performance impact.
>>>
>>> Bill
>>>
>>>
>>> On 8/23/2012 8:21 AM, Mattias Persson wrote:
>>>
>>>> Hi David,
>>>>
>>>> There is nowhere in the code where that node gets deleted. If we refrain
>>>> from that suspicion, could there be something else?