Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> max regionserver handler count


Copy link to this message
-
Re: max regionserver handler count
On Mon, Apr 29, 2013 at 2:49 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> After each zookeeper reconnect, I saw same session Id (0x703e...)
>
> What version of zookeeper are you using ? Can you search zookeeper log for
> this session Id to see what happened ?
>
> Thanks
>

Zookeeper version is 3.4.5, following are the logs from 2 zookeeper
servers. The first one was the first time ever the regionserver connected
to ZK and after that all of them are retries. I doubt the issue is on the
ZK side since I have 3 other services running which run fine with the same
quorum and have no issues.

I feel I am hitting some other bug with HBase when the number of handlers
is increased by a lot. Anyone seen a high number of handlers in any
production deployment out there ?

Thanks,
Viral

===server with first session==2013-04-29 07:40:55,574 [myid:1072011376] - INFO
 [CommitProcessor:1072011376:ZooKeeperServer@595] - Established session
0x703e48a8cfd81be6 with negotiated timeout 40000 for client /
10.155.234.3:53814
EndOfStreamException: Unable to read additional data from client sessionid
0x703e48a8cfd81be6, likely client has closed socket
2013-04-29 07:43:47,737 [myid:1072011376] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for
client /10.155.234.3:53814 which had sessionid 0x703e48a8cfd81be6
2013-04-29 07:46:29,679 [myid:1072011376] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew
session 0x703e48a8cfd81be6 at /10.155.234.3:53831
2013-04-29 07:46:29,679 [myid:1072011376] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:Learner@107] - Revalidating client: 0x703e48a8cfd81be6
2013-04-29 07:46:29,680 [myid:1072011376] - INFO
 [QuorumPeer[myid=1072011376]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@595] -
Established session 0x703e48a8cfd81be6 with negotiated timeout 40000 for
client /10.155.234.3:53831
EndOfStreamException: Unable to read additional data from client sessionid
0x703e48a8cfd81be6, likely client has closed socket
2013-04-29 07:49:10,401 [myid:1072011376] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for
client /10.155.234.3:53831 which had sessionid 0x703e48a8cfd81be6
2013-04-29 07:55:53,441 [myid:1072011376] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew
session 0x703e48a8cfd81be6 at /10.155.234.3:53854
2013-04-29 07:55:53,441 [myid:1072011376] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:Learner@107] - Revalidating client: 0x703e48a8cfd81be6
2013-04-29 07:55:53,442 [myid:1072011376] - INFO
 [QuorumPeer[myid=1072011376]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@595] -
Established session 0x703e48a8cfd81be6 with negotiated timeout 40000 for
client /10.155.234.3:53854
EndOfStreamException: Unable to read additional data from client sessionid
0x703e48a8cfd81be6, likely client has closed socket
2013-04-29 07:58:33,947 [myid:1072011376] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for
client /10.155.234.3:53854 which had sessionid 0x703e48a8cfd81be6

===server with subsequent sessions==2013-04-29 07:44:28,733 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew
session 0x703e48a8cfd81be6 at /10.155.234.3:51353
2013-04-29 07:44:28,734 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@595] - Established session
0x703e48a8cfd81be6 with negotiated timeout 40000 for client /
10.155.234.3:51353
EndOfStreamException: Unable to read additional data from client sessionid
0x703e48a8cfd81be6, likely client has closed socket
2013-04-29 07:46:29,206 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for
client /10.155.234.3:51353 which had sessionid 0x703e48a8cfd81be6
2013-04-29 07:53:52,553 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew
session 0x703e48a8cfd81be6 at /10.155.234.3:51376
2013-04-29 07:53:52,553 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@595] - Established session
0x703e48a8cfd81be6 with negotiated timeout 40000 for client /
10.155.234.3:51376
EndOfStreamException: Unable to read additional data from client sessionid
0x703e48a8cfd81be6, likely client has closed socket
2013-04-29 07:55:53,049 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for
client /10.155.234.3:51376 which had sessionid 0x703e48a8cfd81be6
2013-04-29 08:02:36,000 [myid:54242244232] - INFO
 [SessionTracker:ZooKeeperServer@325] - Expiring session
0x703e48a8cfd81be6, timeout of 40000ms exceeded
2013-04-29 08:02:36,001 [myid:54242244232] - INFO
 [ProcessThread(sid:54242244232 cport:-1)::PrepRequestProcessor@476] -
Processed session termination for sessionid: 0x703e48a8cfd81be6
2013-04-29 08:02:36,283 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew
session 0x703e48a8cfd81be6 at /10.155.234.3:51398
2013-04-29 08:02:36,284 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@588] - Invalid session
0x703e48a8cfd81be6 for client /10.155.234.3:51398, probably expired
2013-04-29 08:02:36,284 [myid:54242244232] - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for
client /10.155.234.3:51398 which had sessionid 0x703e48a8cfd81be6
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB