Hive, mail # user - hiveserver leak zookeeper connections?

hiveserver leak zookeeper connections?

wzc 2013-01-05, 06:51

Hi all:
I am using hive 0.9 and zookeeper 3.3.5, and I set
hive.support.concurrency=true for locking. when I run query through
hiveserver I keep receiving warnings like :

2013-01-05 12:03:56,109 WARN zookeeper.ClientCnxn
(ClientCnxn.java:run(1057)) - Session 0x0 for server
data-dev-01/192.168.2.241:2181, unexpected error, closing socket
connection and a
ttempting reconnect
java.io.IOException: Connection reset by peer
   at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
   at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
   at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
   at sun.nio.ch.IOUtil.read(IOUtil.java:186)
   at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
   at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66
2013-01-05 12:03:56,350 WARN zookeeper.ClientCnxn
(ClientCnxn.java:run(1057)) - Session 0x0 for server
data-dev-02/192.168.2.237:2181, unexpected error, closing socket
connection and a
ttempting reconnect
java.io.IOException: Connection reset by peer
   at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
   at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
   at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
   at sun.nio.ch.IOUtil.read(IOUtil.java:186)
   at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
   at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66)
   at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2013-01-05 12:03:56,476 WARN zookeeper.ClientCnxn
(ClientCnxn.java:run(1057)) - Session 0x0 for server
data-dev-02/192.168.2.237:2181, unexpected error, closing socket
connection and a
ttempting reconnect
java.io.IOException: Connection reset by peer
   at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
   at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
   at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
   at sun.nio.ch.IOUtil.read(IOUtil.java:186)
   at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
   at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66)
   at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
and the query fails. I run 'netstat -anlp | grep 2181' on the server
which hiveserver is running and find many ESTABLISHED connections:

tcp6 0 0 192.168.2.241:50033 192.168.2.241:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:2181 192.168.2.241:50024
ESTABLISHED 22103/java
tcp6 0 0 192.168.2.241:59436 192.168.2.237:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:50015 192.168.2.241:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:59432 192.168.2.237:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:2181 192.168.2.241:50014
ESTABLISHED 22103/java
tcp6 0 0 192.168.2.241:59444 192.168.2.237:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:2181 192.168.2.241:50029
ESTABLISHED 22103/java
tcp6 0 0 192.168.2.241:2181 192.168.2.241:50015
ESTABLISHED 22103/java
tcp6 0 0 192.168.2.241:50031 192.168.2.241:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:50024 192.168.2.241:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:2181 192.168.2.241:49998
ESTABLISHED 22103/java
tcp6 0 0 192.168.2.241:59423 192.168.2.237:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:50021 192.168.2.241:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:59442 192.168.2.237:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:2181 192.168.2.241:50033
ESTABLISHED 22103/java
tcp6 0 0 192.168.2.241:59434 192.168.2.237:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:59426 192.168.2.237:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:59437 192.168.2.237:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:2181 192.168.2.241:50005
ESTABLISHED 22103/java
tcp6 0 0 192.168.2.241:49998 192.168.2.241:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:50029 192.168.2.241:2181
ESTABLISHED 11765/java
tcp6 0 0 192.168.2.241:59439 192.168.2.237:2181
ESTABLISHED 11765/java
when I turn off hive.support.concurrency option it's ok.

I think maybe hiveserver is leaking zookeeper connections. Can anyone
give me some hint?
Thanks.

Re: hiveserver leak zookeeper connections?

Ashutosh Chauhan 2013-01-05, 19:09

We have found one source of leak which we think we have fixed.
https://issues.apache.org/jira/browse/HIVE-3723 This is a part of hive-0.10
RC0 http://people.apache.org/~hashutosh/hive-0.10.0-rc0/ You can try hive
from there and see if this fixes this connection leaking problem.

Thanks,
Ashutosh
On Fri, Jan 4, 2013 at 10:51 PM, wzc <[EMAIL PROTECTED]> wrote:

> Hi all:
> I am using hive 0.9 and zookeeper 3.3.5, and I set
> hive.support.concurrency=true for locking. when I run query through
> hiveserver I keep receiving warnings like :
>
> 2013-01-05 12:03:56,109 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-01/192.168.2.241:2181, unexpected error, closing socket connection and a
> ttempting reconnect
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66
> 2013-01-05 12:03:56,350 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-02/192.168.2.237:2181, unexpected error, closing socket connection and a
> ttempting reconnect
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2013-01-05 12:03:56,476 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-02/192.168.2.237:2181, unexpected error, closing socket connection and a
> ttempting reconnect
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>
>
> and the query fails. I run 'netstat -anlp | grep 2181' on the server which hiveserver is running and find many ESTABLISHED connections:
>
> tcp6 0 0 192.168.2.241:50033 192.168.2.241:2181 ESTABLISHED 11765/java
> tcp6 0 0 192.168.2.241:2181 192.168.2.241:50024 ESTABLISHED 22103/java
> tcp6 0 0 192.168.2.241:59436 192.168.2.237:2181 ESTABLISHED 11765/java
> tcp6 0 0 192.168.2.241:50015 192.168.2.241:2181 ESTABLISHED 11765/java
> tcp6 0 0 192.168.2.241:59432 192.168.2.237:2181 ESTABLISHED 11765/java
> tcp6 0 0 192.168.2.241:2181 192.168.2.241:50014 ESTABLISHED 22103/java
> tcp6 0 0 192.168.2.241:59444 192.168.2.237:2181 ESTABLISHED 11765/java
> tcp6 0 0 192.168.2.241:2181 192.168.2.241:50029 ESTABLISHED 22103/java
> tcp6 0 0 192.168.2.241:2181 192.168.2.241:50015 ESTABLISHED 22103/java
> tcp6 0 0 192.168.2.241:50031 192.168.2.241:2181 ESTABLISHED 11765/java

Re: hiveserver leak zookeeper connections?

wzc 2013-01-07, 13:48

@ Ashutosh:
I patched HIVE-3723 with hive 0.9 but it didn't work, I would try using
hive 0.10.0.
Thanks~
2013/1/6 Ashutosh Chauhan <[EMAIL PROTECTED]>

> We have found one source of leak which we think we have fixed.
> https://issues.apache.org/jira/browse/HIVE-3723 This is a part of
> hive-0.10 RC0 http://people.apache.org/~hashutosh/hive-0.10.0-rc0/ You
> can try hive from there and see if this fixes this connection leaking
> problem.
>
> Thanks,
> Ashutosh
>
>
> On Fri, Jan 4, 2013 at 10:51 PM, wzc <[EMAIL PROTECTED]> wrote:
>
>> Hi all:
>> I am using hive 0.9 and zookeeper 3.3.5, and I set
>> hive.support.concurrency=true for locking. when I run query through
>> hiveserver I keep receiving warnings like :
>>
>> 2013-01-05 12:03:56,109 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-01/192.168.2.241:2181, unexpected error, closing socket connection and a
>> ttempting reconnect
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66
>> 2013-01-05 12:03:56,350 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-02/192.168.2.237:2181, unexpected error, closing socket connection and a
>> ttempting reconnect
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>> 2013-01-05 12:03:56,476 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-02/192.168.2.237:2181, unexpected error, closing socket connection and a
>> ttempting reconnect
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>>
>>
>> and the query fails. I run 'netstat -anlp | grep 2181' on the server which hiveserver is running and find many ESTABLISHED connections:
>>
>> tcp6 0 0 192.168.2.241:50033 192.168.2.241:2181 ESTABLISHED 11765/java
>> tcp6 0 0 192.168.2.241:2181 192.168.2.241:50024 ESTABLISHED 22103/java
>> tcp6 0 0 192.168.2.241:59436 192.168.2.237:2181 ESTABLISHED 11765/java
>> tcp6 0 0 192.168.2.241:50015 192.168.2.241:2181 ESTABLISHED 11765/java
>> tcp6 0 0 192.168.2.241:59432 192.168.2.237:2181 ESTABLISHED 11765/java
>> tcp6 0 0 192.168.2.241:2181 192.168.2.241:50014 ESTABLISHED 22103/java
>> tcp6 0 0 192.168.2.241:59444 192.168.2.237:2181 ESTABLISHED 11765/java
>> tcp6 0 0 192.168.2.241:2181 192.168.2.241:50029 ESTABLISHED 22103/java

Re: hiveserver leak zookeeper connections?

Edward Capriolo 2013-01-07, 16:35

I had a similar problem with hive 0.7.0 and zk-locking. I turned it off.
Just too much trouble.

On Mon, Jan 7, 2013 at 8:48 AM, wzc <[EMAIL PROTECTED]> wrote:

> @ Ashutosh:
> I patched HIVE-3723 with hive 0.9 but it didn't work, I would try using
> hive 0.10.0.
> Thanks~
>
>
> 2013/1/6 Ashutosh Chauhan <[EMAIL PROTECTED]>
>
>> We have found one source of leak which we think we have fixed.
>> https://issues.apache.org/jira/browse/HIVE-3723 This is a part of
>> hive-0.10 RC0 http://people.apache.org/~hashutosh/hive-0.10.0-rc0/ You
>> can try hive from there and see if this fixes this connection leaking
>> problem.
>>
>> Thanks,
>> Ashutosh
>>
>>
>> On Fri, Jan 4, 2013 at 10:51 PM, wzc <[EMAIL PROTECTED]> wrote:
>>
>>> Hi all:
>>> I am using hive 0.9 and zookeeper 3.3.5, and I set
>>> hive.support.concurrency=true for locking. when I run query through
>>> hiveserver I keep receiving warnings like :
>>>
>>> 2013-01-05 12:03:56,109 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-01/192.168.2.241:2181, unexpected error, closing socket connection and a
>>> ttempting reconnect
>>> java.io.IOException: Connection reset by peer
>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>>> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66
>>> 2013-01-05 12:03:56,350 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-02/192.168.2.237:2181, unexpected error, closing socket connection and a
>>> ttempting reconnect
>>> java.io.IOException: Connection reset by peer
>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>>> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66)
>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>>> 2013-01-05 12:03:56,476 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0x0 for server data-dev-02/192.168.2.237:2181, unexpected error, closing socket connection and a
>>> ttempting reconnect
>>> java.io.IOException: Connection reset by peer
>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)
>>> at sun.nio.ch.IOUtil.read(IOUtil.java:186)
>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:66)
>>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:291)
>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>>>
>>>
>>> and the query fails. I run 'netstat -anlp | grep 2181' on the server which hiveserver is running and find many ESTABLISHED connections:
>>>
>>> tcp6 0 0 192.168.2.241:50033 192.168.2.241:2181 ESTABLISHED 11765/java
>>> tcp6 0 0 192.168.2.241:2181 192.168.2.241:50024 ESTABLISHED 22103/java
>>> tcp6 0 0 192.168.2.241:59436 192.168.2.237:2181 ESTABLISHED 11765/java
>>> tcp6 0 0 192.168.2.241:50015 192.168.2.241:2181 ESTABLISHED 11765/java
>>> tcp6 0 0 192.168.2.241:59432 192.168.2.237:2181 ESTABLISHED 11765/java
>