Hi Everyone,

 

Our customers are recently complaining about zookeeper client session
timeouts.

When analyzed its found that timeouts are due to heavy GC activity on
Clients.

 

So, they wanted to increase the session timeouts to 3 minutes which requires
the ticktime to be increased to atleast 9 seconds.

 

I wanted to know the impacts of increasing the ticktime to 10 seconds.

 

Following is my understanding of impacts. Correct me if my understanding is
wrong.

1) Leader election and syncing on startup may be delayed.

2) Watches on ephemeral nodes will get notifications only after session
timeout period.

 

Still I have some open questions w.r.to increasing ticktime.

1) Does this impact any performance?

2) Any other impacts?

3) Any other configurations to be tuned along with this?

 

--

Thanks,

Laxman

On Wed, Sep 14, 2011 at 3:23 AM, Laxman <[EMAIL PROTECTED]> wrote:
> Our customers are recently complaining about zookeeper client session
> timeouts.
>
> When analyzed its found that timeouts are due to heavy GC activity on
> Clients.
>
> So, they wanted to increase the session timeouts to 3 minutes which requires
> the ticktime to be increased to atleast 9 seconds.
>

I assume you've talked to them about fixing their gc issue at some
point? Rather than band-aiding it? :-)

I don't think you need to touch the tickTime here. See "maxSessionTimeout" here:
http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConfiguration
(hbase has a similar issue, that's why we added this feature initially).

Really though they should fix the gc issue - setting the session
timeout higher means that their sessions will be expiring only after a
much longer time. Any reliance on this - such as for leader election
fail-over, will now take much more time. In your example it would take
3 minutes for the other clients to notice when a client with the
particular session has become unavailable. Perhaps it doesn't matter
in your use case, but you are now likely to get complains about ZK not
being responsive. ;-)

Regards,

Patrick

Session timeout is negotiated though eh? So if only the clients that need the long GC session expiration time set their client session timeout to MAX_INT, other clients can keep it at a lower number and not have a problem. That's how we have resolved the issue. It does mean that if you have clients writing directly to the ZK client (instead of through a wrapper you provide), you need to encourage people not to use MAX_INT in their default client setup unless they need the long timeout.

C

-----Original Message-----
From: Patrick Hunt [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 14, 2011 1:44 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: Impacts of increasing ZooKeeper ticktime

On Wed, Sep 14, 2011 at 3:23 AM, Laxman <[EMAIL PROTECTED]> wrote:
> Our customers are recently complaining about zookeeper client session
> timeouts.
>
> When analyzed its found that timeouts are due to heavy GC activity on
> Clients.
>
> So, they wanted to increase the session timeouts to 3 minutes which requires
> the ticktime to be increased to atleast 9 seconds.
>

I assume you've talked to them about fixing their gc issue at some
point? Rather than band-aiding it? :-)

I don't think you need to touch the tickTime here. See "maxSessionTimeout" here:
http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConfiguration
(hbase has a similar issue, that's why we added this feature initially).

Really though they should fix the gc issue - setting the session
timeout higher means that their sessions will be expiring only after a
much longer time. Any reliance on this - such as for leader election
fail-over, will now take much more time. In your example it would take
3 minutes for the other clients to notice when a client with the
particular session has become unavailable. Perhaps it doesn't matter
in your use case, but you are now likely to get complains about ZK not
being responsive. ;-)

Regards,

Patrick

Pat & Camille, Thanks for your response.

> I don't think you need to touch the tickTime here. See "maxSessionTimeout"
> here:
> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConf
> iguration
> (hbase has a similar issue, that's why we added this feature initially).

I've tried this maxSessionTimeout. This satisfies my customer's requirement.
I also agree that, we still need to fix the GC issue. This has been
discussed with our customer and "maxSessionTimeout" is provided to unblock
them temporarily.

>It does mean that if you
> have clients writing directly to the ZK client (instead of through a
> wrapper you provide), you need to encourage people not to use MAX_INT in
> their default client setup unless they need the long timeout.

This is clear and in fact we are using the session timeout value of 30
seconds only for many use cases like Namenode HA.

Thanks again for your quick response.

> -----Original Message-----
> From: Fournier, Camille F. [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, September 14, 2011 11:48 PM
> To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
> Subject: RE: Impacts of increasing ZooKeeper ticktime
>
> Session timeout is negotiated though eh? So if only the clients that need
> the long GC session expiration time set their client session timeout to
> MAX_INT, other clients can keep it at a lower number and not have a
> problem. That's how we have resolved the issue. It does mean that if you
> have clients writing directly to the ZK client (instead of through a
> wrapper you provide), you need to encourage people not to use MAX_INT in
> their default client setup unless they need the long timeout.
>
> C
>
> -----Original Message-----
> From: Patrick Hunt [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, September 14, 2011 1:44 PM
> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> Subject: Re: Impacts of increasing ZooKeeper ticktime
>
> On Wed, Sep 14, 2011 at 3:23 AM, Laxman <[EMAIL PROTECTED]> wrote:
> > Our customers are recently complaining about zookeeper client session
> > timeouts.
> >
> > When analyzed its found that timeouts are due to heavy GC activity on
> > Clients.
> >
> > So, they wanted to increase the session timeouts to 3 minutes which
> requires
> > the ticktime to be increased to atleast 9 seconds.
> >
>
> I assume you've talked to them about fixing their gc issue at some
> point? Rather than band-aiding it? :-)
>
> I don't think you need to touch the tickTime here. See "maxSessionTimeout"
> here:
> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConf
> iguration
> (hbase has a similar issue, that's why we added this feature initially).
>
> Really though they should fix the gc issue - setting the session
> timeout higher means that their sessions will be expiring only after a
> much longer time. Any reliance on this - such as for leader election
> fail-over, will now take much more time. In your example it would take
> 3 minutes for the other clients to notice when a client with the
> particular session has become unavailable. Perhaps it doesn't matter
> in your use case, but you are now likely to get complains about ZK not
> being responsive. ;-)
>
> Regards,
>
> Patrick

On Wed, Sep 14, 2011 at 9:30 PM, Laxman <[EMAIL PROTECTED]> wrote:
> Pat & Camille, Thanks for your response.
>
>> I don't think you need to touch the tickTime here. See "maxSessionTimeout"
>> here:
>> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConf
>> iguration
>> (hbase has a similar issue, that's why we added this feature initially).
>
> I've tried this maxSessionTimeout. This satisfies my customer's requirement.
> I also agree that, we still need to fix the GC issue. This has been
> discussed with our customer and "maxSessionTimeout" is provided to unblock
> them temporarily.
>
>>It does mean that if you
>> have clients writing directly to the ZK client (instead of through a
>> wrapper you provide), you need to encourage people not to use MAX_INT in
>> their default client setup unless they need the long timeout.
>
> This is clear and in fact we are using the session timeout value of 30
> seconds only for many use cases like Namenode HA.
>

That's true (see Camille's response, I missed highlighting this in my
response, but it's absolutely the case).

> Thanks again for your quick response.

NP. Good Luck.

Patrick

>
>> -----Original Message-----
>> From: Fournier, Camille F. [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, September 14, 2011 11:48 PM
>> To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
>> Subject: RE: Impacts of increasing ZooKeeper ticktime
>>
>> Session timeout is negotiated though eh? So if only the clients that need
>> the long GC session expiration time set their client session timeout to
>> MAX_INT, other clients can keep it at a lower number and not have a
>> problem. That's how we have resolved the issue. It does mean that if you
>> have clients writing directly to the ZK client (instead of through a
>> wrapper you provide), you need to encourage people not to use MAX_INT in
>> their default client setup unless they need the long timeout.
>>
>> C
>>
>> -----Original Message-----
>> From: Patrick Hunt [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, September 14, 2011 1:44 PM
>> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
>> Subject: Re: Impacts of increasing ZooKeeper ticktime
>>
>> On Wed, Sep 14, 2011 at 3:23 AM, Laxman <[EMAIL PROTECTED]> wrote:
>> > Our customers are recently complaining about zookeeper client session
>> > timeouts.
>> >
>> > When analyzed its found that timeouts are due to heavy GC activity on
>> > Clients.
>> >
>> > So, they wanted to increase the session timeouts to 3 minutes which
>> requires
>> > the ticktime to be increased to atleast 9 seconds.
>> >
>>
>> I assume you've talked to them about fixing their gc issue at some
>> point? Rather than band-aiding it? :-)
>>
>> I don't think you need to touch the tickTime here. See "maxSessionTimeout"
>> here:
>> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConf
>> iguration
>> (hbase has a similar issue, that's why we added this feature initially).
>>
>> Really though they should fix the gc issue - setting the session
>> timeout higher means that their sessions will be expiring only after a
>> much longer time. Any reliance on this - such as for leader election
>> fail-over, will now take much more time. In your example it would take
>> 3 minutes for the other clients to notice when a client with the
>> particular session has become unavailable. Perhaps it doesn't matter
>> in your use case, but you are now likely to get complains about ZK not
>> being responsive. ;-)
>>
>> Regards,
>>
>> Patrick
>
>