Hi All,

I am using Zookeeper 3.4.3 sever and my setup is as follows:

I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
on my laptop.
Then I start 12 Zookeeper client processes on the same machine which start
updates on Zookeeper database.
After running test for a while, sometimes some of the client process got
disconnected from Zookeeper sever with following error:
*Exceeded deadline by 16ms* and then says session timeout.
Client connects with timeout value of 10000ms.

Zookeeper server configuration is as follows:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
clientPort=2181
server.1=192.168.1.234:2888:3888
server.2=192.168.1.235:2888:3888
server.3=192.168.1.236:2888:3888

Does anybody facing similar problem? Please let me know if there anythin
wrong wit configuration.

Thanks & Regards,
Deepak

Hello,

Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
on watch callback event or just simply debug information print on
screen showing "Exceeded deadline by 16ms"?

Thanks.
2012/5/17 Deepak Jagtap <[EMAIL PROTECTED]>:
> Hi All,
>
> I am using Zookeeper 3.4.3 sever and my setup is as follows:
>
> I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
> on my laptop.
> Then I start 12 Zookeeper client processes on the same machine which start
> updates on Zookeeper database.
> After running test for a while, sometimes some of the client process got
> disconnected from Zookeeper sever with following error:
> *Exceeded deadline by 16ms* and then says session timeout.
> Client connects with timeout value of 10000ms.
>
> Zookeeper server configuration is as follows:
> tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/tmp/zookeeper
> clientPort=2181
> server.1=192.168.1.234:2888:3888
> server.2=192.168.1.235:2888:3888
> server.3=192.168.1.236:2888:3888
>
> Does anybody facing similar problem? Please let me know if there anythin
> wrong wit configuration.
>
> Thanks & Regards,
> Deepak

Hi,

Thanks for quick reply!
I am seeing sequence of "exceeded deadline" message then  connection timed
out error messages.
My code has some usleep() calls, when a sequence of zoo_get and zoo_set
returns ZBADVERSION error due to race condition,
it invokes usleep() for random time(between 5ms to 100ms) and then retry
the same operation. Does this may be cause any issues?

Please find following log dump:

2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 14ms
2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 11ms
.
.
2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 11ms
2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
Exceeded deadline by 11ms
2012-05-17
11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg@1644:
Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed out):
connection timed out (exceeded timeout by 0ms)
operation timeout
operation timeout
2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
initiated connection to server [192.168.1.234:2181]
Thanks & Regards,
Deepak

On Wed, May 16, 2012 at 6:30 PM, 李赫元 <[EMAIL PROTECTED]> wrote:

> Hello,
>
> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
> on watch callback event or just simply debug information print on
> screen showing "Exceeded deadline by 16ms"?
>
> Thanks.
>
>
> 2012/5/17 Deepak Jagtap <[EMAIL PROTECTED]>:
> > Hi All,
> >
> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
> >
> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
> > on my laptop.
> > Then I start 12 Zookeeper client processes on the same machine which
> start
> > updates on Zookeeper database.
> > After running test for a while, sometimes some of the client process got
> > disconnected from Zookeeper sever with following error:
> > *Exceeded deadline by 16ms* and then says session timeout.
> > Client connects with timeout value of 10000ms.
> >
> > Zookeeper server configuration is as follows:
> > tickTime=2000
> > initLimit=10
> > syncLimit=5
> > dataDir=/tmp/zookeeper
> > clientPort=2181
> > server.1=192.168.1.234:2888:3888
> > server.2=192.168.1.235:2888:3888
> > server.3=192.168.1.236:2888:3888
> >
> > Does anybody facing similar problem? Please let me know if there anythin
> > wrong wit configuration.
> >
> > Thanks & Regards,
> > Deepak
>

As said by @phunt, the exceed deadline is client's warning.
It's different from SESSION_TIMEOUT, which require re-init handle and
reset all watcher.
The server may be a little overload.

2012/5/18 Deepak Jagtap <[EMAIL PROTECTED]>:
> Hi,
>
> Thanks for quick reply!
> I am seeing sequence of "exceeded deadline" message then  connection timed
> out error messages.
> My code has some usleep() calls, when a sequence of zoo_get and zoo_set
> returns ZBADVERSION error due to race condition,
> it invokes usleep() for random time(between 5ms to 100ms) and then retry
> the same operation. Does this may be cause any issues?
>
> Please find following log dump:
>
> 2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 14ms
> 2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> .
> .
> 2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> 2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> 2012-05-17
> 11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg@1644:
> Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed out):
> connection timed out (exceeded timeout by 0ms)
> operation timeout
> operation timeout
> 2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
> initiated connection to server [192.168.1.234:2181]
>
>
> Thanks & Regards,
> Deepak
>
> On Wed, May 16, 2012 at 6:30 PM, 李赫元 <[EMAIL PROTECTED]> wrote:
>
>> Hello,
>>
>> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
>> on watch callback event or just simply debug information print on
>> screen showing "Exceeded deadline by 16ms"?
>>
>> Thanks.
>>
>>
>> 2012/5/17 Deepak Jagtap <[EMAIL PROTECTED]>:
>> > Hi All,
>> >
>> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
>> >
>> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
>> > on my laptop.
>> > Then I start 12 Zookeeper client processes on the same machine which
>> start
>> > updates on Zookeeper database.
>> > After running test for a while, sometimes some of the client process got
>> > disconnected from Zookeeper sever with following error:
>> > *Exceeded deadline by 16ms* and then says session timeout.
>> > Client connects with timeout value of 10000ms.
>> >
>> > Zookeeper server configuration is as follows:
>> > tickTime=2000
>> > initLimit=10
>> > syncLimit=5
>> > dataDir=/tmp/zookeeper
>> > clientPort=2181
>> > server.1=192.168.1.234:2888:3888
>> > server.2=192.168.1.235:2888:3888
>> > server.3=192.168.1.236:2888:3888
>> >
>> > Does anybody facing similar problem? Please let me know if there anythin
>> > wrong wit configuration.
>> >
>> > Thanks & Regards,
>> > Deepak
>>

The deadline exceeded messages are generated by select - the client
will select(timeout). If the requested timeout is exceeded by more
than 10ms then that message is printed. Often this will happen in a
virtualized client environment (not the case here from what you are
saying), or an overloaded OS, etc...

You are running 12 client processes, where the client code expects a
10second session timeout (so heartbeat responses w/in 3.3 seconds)
along three vms running three servers all on the same laptop, it's
likely that you are overloaded.

Patrick

On Thu, May 17, 2012 at 11:36 AM, Deepak Jagtap <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Thanks for quick reply!
> I am seeing sequence of "exceeded deadline" message then  connection timed
> out error messages.
> My code has some usleep() calls, when a sequence of zoo_get and zoo_set
> returns ZBADVERSION error due to race condition,
> it invokes usleep() for random time(between 5ms to 100ms) and then retry
> the same operation. Does this may be cause any issues?
>
> Please find following log dump:
>
> 2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 14ms
> 2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> .
> .
> 2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> 2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest@1557:
> Exceeded deadline by 11ms
> 2012-05-17
> 11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg@1644:
> Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed out):
> connection timed out (exceeded timeout by 0ms)
> operation timeout
> operation timeout
> 2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
> initiated connection to server [192.168.1.234:2181]
>
>
> Thanks & Regards,
> Deepak
>
> On Wed, May 16, 2012 at 6:30 PM, 李赫元 <[EMAIL PROTECTED]> wrote:
>
>> Hello,
>>
>> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
>> on watch callback event or just simply debug information print on
>> screen showing "Exceeded deadline by 16ms"?
>>
>> Thanks.
>>
>>
>> 2012/5/17 Deepak Jagtap <[EMAIL PROTECTED]>:
>> > Hi All,
>> >
>> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
>> >
>> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual machines)
>> > on my laptop.
>> > Then I start 12 Zookeeper client processes on the same machine which
>> start
>> > updates on Zookeeper database.
>> > After running test for a while, sometimes some of the client process got
>> > disconnected from Zookeeper sever with following error:
>> > *Exceeded deadline by 16ms* and then says session timeout.
>> > Client connects with timeout value of 10000ms.
>> >
>> > Zookeeper server configuration is as follows:
>> > tickTime=2000
>> > initLimit=10
>> > syncLimit=5
>> > dataDir=/tmp/zookeeper
>> > clientPort=2181
>> > server.1=192.168.1.234:2888:3888
>> > server.2=192.168.1.235:2888:3888
>> > server.3=192.168.1.236:2888:3888
>> >
>> > Does anybody facing similar problem? Please let me know if there anythin
>> > wrong wit configuration.
>> >
>> > Thanks & Regards,
>> > Deepak
>>

Thanks Patrick, that helped!

Best Regards,
Deepak

On Thu, May 17, 2012 at 3:23 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:

> The deadline exceeded messages are generated by select - the client
> will select(timeout). If the requested timeout is exceeded by more
> than 10ms then that message is printed. Often this will happen in a
> virtualized client environment (not the case here from what you are
> saying), or an overloaded OS, etc...
>
> You are running 12 client processes, where the client code expects a
> 10second session timeout (so heartbeat responses w/in 3.3 seconds)
> along three vms running three servers all on the same laptop, it's
> likely that you are overloaded.
>
> Patrick
>
> On Thu, May 17, 2012 at 11:36 AM, Deepak Jagtap <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > Thanks for quick reply!
> > I am seeing sequence of "exceeded deadline" message then  connection
> timed
> > out error messages.
> > My code has some usleep() calls, when a sequence of zoo_get and zoo_set
> > returns ZBADVERSION error due to race condition,
> > it invokes usleep() for random time(between 5ms to 100ms) and then retry
> > the same operation. Does this may be cause any issues?
> >
> > Please find following log dump:
> >
> > 2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
> @1557:
> > Exceeded deadline by 14ms
> > 2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
> @1557:
> > Exceeded deadline by 11ms
> > .
> > .
> > 2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
> @1557:
> > Exceeded deadline by 11ms
> > 2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
> @1557:
> > Exceeded deadline by 11ms
> > 2012-05-17
> > 11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg
> @1644:
> > Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed
> out):
> > connection timed out (exceeded timeout by 0ms)
> > operation timeout
> > operation timeout
> > 2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
> > initiated connection to server [192.168.1.234:2181]
> >
> >
> > Thanks & Regards,
> > Deepak
> >
> > On Wed, May 16, 2012 at 6:30 PM, 李赫元 <[EMAIL PROTECTED]> wrote:
> >
> >> Hello,
> >>
> >> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
> >> on watch callback event or just simply debug information print on
> >> screen showing "Exceeded deadline by 16ms"?
> >>
> >> Thanks.
> >>
> >>
> >> 2012/5/17 Deepak Jagtap <[EMAIL PROTECTED]>:
> >> > Hi All,
> >> >
> >> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
> >> >
> >> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual
> machines)
> >> > on my laptop.
> >> > Then I start 12 Zookeeper client processes on the same machine which
> >> start
> >> > updates on Zookeeper database.
> >> > After running test for a while, sometimes some of the client process
> got
> >> > disconnected from Zookeeper sever with following error:
> >> > *Exceeded deadline by 16ms* and then says session timeout.
> >> > Client connects with timeout value of 10000ms.
> >> >
> >> > Zookeeper server configuration is as follows:
> >> > tickTime=2000
> >> > initLimit=10
> >> > syncLimit=5
> >> > dataDir=/tmp/zookeeper
> >> > clientPort=2181
> >> > server.1=192.168.1.234:2888:3888
> >> > server.2=192.168.1.235:2888:3888
> >> > server.3=192.168.1.236:2888:3888
> >> >
> >> > Does anybody facing similar problem? Please let me know if there
> anythin
> >> > wrong wit configuration.
> >> >
> >> > Thanks & Regards,
> >> > Deepak
> >>
>

NP, good luck.

Patrick

On Thu, May 17, 2012 at 4:01 PM, Deepak Jagtap <[EMAIL PROTECTED]> wrote:
> Thanks Patrick, that helped!
>
> Best Regards,
> Deepak
>
> On Thu, May 17, 2012 at 3:23 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>
>> The deadline exceeded messages are generated by select - the client
>> will select(timeout). If the requested timeout is exceeded by more
>> than 10ms then that message is printed. Often this will happen in a
>> virtualized client environment (not the case here from what you are
>> saying), or an overloaded OS, etc...
>>
>> You are running 12 client processes, where the client code expects a
>> 10second session timeout (so heartbeat responses w/in 3.3 seconds)
>> along three vms running three servers all on the same laptop, it's
>> likely that you are overloaded.
>>
>> Patrick
>>
>> On Thu, May 17, 2012 at 11:36 AM, Deepak Jagtap <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > Thanks for quick reply!
>> > I am seeing sequence of "exceeded deadline" message then  connection
>> timed
>> > out error messages.
>> > My code has some usleep() calls, when a sequence of zoo_get and zoo_set
>> > returns ZBADVERSION error due to race condition,
>> > it invokes usleep() for random time(between 5ms to 100ms) and then retry
>> > the same operation. Does this may be cause any issues?
>> >
>> > Please find following log dump:
>> >
>> > 2012-05-17 11:10:12,910:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
>> @1557:
>> > Exceeded deadline by 14ms
>> > 2012-05-17 11:10:13,200:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
>> @1557:
>> > Exceeded deadline by 11ms
>> > .
>> > .
>> > 2012-05-17 11:10:13,270:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
>> @1557:
>> > Exceeded deadline by 11ms
>> > 2012-05-17 11:10:13,745:6709(0x7f12d52d2700):ZOO_WARN@zookeeper_interest
>> @1557:
>> > Exceeded deadline by 11ms
>> > 2012-05-17
>> > 11:10:14,164:6709(0x7f12d52d2700):ZOO_ERROR@handle_socket_error_msg
>> @1644:
>> > Socket [192.168.1.236:2181] zk retcode=-7, errno=110(Connection timed
>> out):
>> > connection timed out (exceeded timeout by 0ms)
>> > operation timeout
>> > operation timeout
>> > 2012-05-17 11:10:14,164:6709(0x7f12d52d2700):ZOO_INFO@check_events@1701:
>> > initiated connection to server [192.168.1.234:2181]
>> >
>> >
>> > Thanks & Regards,
>> > Deepak
>> >
>> > On Wed, May 16, 2012 at 6:30 PM, 李赫元 <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hello,
>> >>
>> >> Did you get the "ZOO_SESSION_EVENT" with "ZOO_EXPIRED_SESSION_STATE"
>> >> on watch callback event or just simply debug information print on
>> >> screen showing "Exceeded deadline by 16ms"?
>> >>
>> >> Thanks.
>> >>
>> >>
>> >> 2012/5/17 Deepak Jagtap <[EMAIL PROTECTED]>:
>> >> > Hi All,
>> >> >
>> >> > I am using Zookeeper 3.4.3 sever and my setup is as follows:
>> >> >
>> >> > I have a Quorum of 3 Zookeepers servers (installed on 3 virtual
>> machines)
>> >> > on my laptop.
>> >> > Then I start 12 Zookeeper client processes on the same machine which
>> >> start
>> >> > updates on Zookeeper database.
>> >> > After running test for a while, sometimes some of the client process
>> got
>> >> > disconnected from Zookeeper sever with following error:
>> >> > *Exceeded deadline by 16ms* and then says session timeout.
>> >> > Client connects with timeout value of 10000ms.
>> >> >
>> >> > Zookeeper server configuration is as follows:
>> >> > tickTime=2000
>> >> > initLimit=10
>> >> > syncLimit=5
>> >> > dataDir=/tmp/zookeeper
>> >> > clientPort=2181
>> >> > server.1=192.168.1.234:2888:3888
>> >> > server.2=192.168.1.235:2888:3888
>> >> > server.3=192.168.1.236:2888:3888
>> >> >
>> >> > Does anybody facing similar problem? Please let me know if there
>> anythin
>> >> > wrong wit configuration.
>> >> >
>> >> > Thanks & Regards,
>> >> > Deepak
>> >>
>>