Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Zookeeper sync timeout issues


Copy link to this message
-
Re: Zookeeper sync timeout issues
Jon,
 Whats the size of the snapshot?

And what are the configs for:

1) initLimit
2) syncLimit
3) tickTime
?

thanks
mahadev

On Mon, Oct 24, 2011 at 11:09 AM, Jon King <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> It looks like one of our ZK quorum servers cannot sync with the leader
> anymore.  The leader logs show "Read timed out" errors and the follower is
> showing a "Broken pipe" at the same time.
>
> Follower logs
>
> 2011-10-24 11:53:23,110 - INFO  [QuorumPeer:/0.0.0.0:2181:FileSnap@82] -
> Reading snapshot /var/zookeeper/version-2/snapshot.10000de07
> 2011-10-24 11:53:32,792 - WARN  [QuorumPeer:/0.0.0.0:2181:QuorumPeer@497]
> - Unable to load database
> java.io.IOException: Transaction log:
> /var/zookeeper/version-2/log.10000de08 has invalid magic number 0 !> 1514884167
>         at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:510)
>         at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:527)
>         at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:493)
>         at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:475)
>         at
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:454)
>         at
> org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:325)
>         at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:126)
>         at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
>         at
> org.apache.zookeeper.server.quorum.QuorumPeer.getLastLoggedZxid(QuorumPeer.java:493)
>         at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:69)
>         at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2011-10-24 11:53:32,793 - INFO  [QuorumPeer:/0.0.0.0:2181:Learner@294] -
> Getting a snapshot from leader
> 2011-10-24 11:54:19,716 - INFO  [QuorumPeer:/0.0.0.0:2181:Learner@325] -
> Setting leader epoch 1
> 2011-10-24 11:54:19,717 - INFO  [QuorumPeer:/0.0.0.0:2181
> :FileTxnSnapLog@208] - Snapshotting: 10000de0d
> 2011-10-24 11:54:44,412 - WARN  [QuorumPeer:/0.0.0.0:2181:Follower@82] -
> Exception when following the leader
> java.net.SocketException: Broken pipe
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at java.net.SocketOutputStream.socketWrite(Unknown Source)
>         at java.net.SocketOutputStream.write(Unknown Source)
>         at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
>         at java.io.BufferedOutputStream.flush(Unknown Source)
>         at
> org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:134)
>         at
> org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:418)
>         at
> org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:108)
>         at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:79)
>         at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2011-10-24 11:54:45,784 - INFO  [QuorumPeer:/0.0.0.0:2181:Follower@165] -
> shutdown called
> java.lang.Exception: shutdown Follower
>         at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
>         at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2011-10-24 11:54:45,785 - INFO  [QuorumPeer:/0.0.0.0:2181
> :FinalRequestProcessor@378] - shutdown of request processor complete
>
>
> Leader Logs
>
> 2011-10-24 11:53:13,626 - INFO  [WorkerReceiver
> Thread:FastLeaderElection@496] - Notification: 3 (n.leader), -1 (n.zxid),
> 2 (n.round), LOOKING (n.state), 3 (n.sid), LEADING (my state)
> 2011-10-24 11:53:23,109 - INFO  [LearnerHandler-/10.3.4.156:41450
> :LearnerHandler@249] - Follower sid: 3 : info :
> org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@783c342b