Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - Hive Metastore Server 0.9 Connection Reset and Connection Timeout errors


+
agateaaa 2013-07-29, 17:43
+
Nitin Pawar 2013-07-29, 18:02
+
agateaaa 2013-07-29, 18:29
Copy link to this message
-
Re: Hive Metastore Server 0.9 Connection Reset and Connection Timeout errors
agateaaa 2013-07-30, 00:22
Looking at the hive metastore server logs see errors like these:

2013-07-26 06:34:52,853 ERROR server.TThreadPoolServer
(TThreadPoolServer.java:run(182)) - Error occurred during processing of
message.
java.lang.NullPointerException
        at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.setIpAddress(TUGIBasedProcessor.java:183)
        at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:79)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

approx same time as we see timeout or connection reset errors.

Dont know if this is the cause or the side affect of he connection
timeout/connection reset errors. Does anybody have any pointers or
suggestions ?

Thanks
On Mon, Jul 29, 2013 at 11:29 AM, agateaaa <[EMAIL PROTECTED]> wrote:

> Thanks Nitin!
>
> We have simiar setup (identical hcatalog and hive server versions) on a
> another production environment and dont see any errors (its been running ok
> for a few months)
>
> Unfortunately we wont be able to move to hcat 0.5 and hive 0.11 or hive
> 0.10 soon.
>
> I did see that the last time we ran into this problem doing a netstat-ntp
> | grep ":10000" see that server was holding on to one socket connection in
> CLOSE_WAIT state for a long time
>  (hive metastore server is running on port 10000). Dont know if thats
> relevant here or not
>
> Can you suggest any hive configuration settings we can tweak or networking
> tools/tips, we can use to narrow this down ?
>
> Thanks
> Agateaaa
>
>
>
>
> On Mon, Jul 29, 2013 at 11:02 AM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> Is there any chance you can do a update on test environment with hcat-0.5
>> and hive-0(11 or 10) and see if you can reproduce the issue?
>>
>> We used to see this error when there was load on hcat server or some
>> network issue connecting to the server(second one was rare occurrence)
>>
>>
>> On Mon, Jul 29, 2013 at 11:13 PM, agateaaa <[EMAIL PROTECTED]> wrote:
>>
>>> Hi All:
>>>
>>> We are running into frequent problem using HCatalog 0.4.1 (HIve Metastore
>>> Server 0.9) where we get connection reset or connection timeout errors.
>>>
>>> The hive metastore server has been allocated enough (12G) memory.
>>>
>>> This is a critical problem for us and would appreciate if anyone has any
>>> pointers.
>>>
>>> We did add a retry logic in our client, which seems to help, but I am
>>> just
>>> wondering how can we narrow down to the root cause
>>> of this problem. Could this be a hiccup in networking which causes the
>>> hive
>>> server to get into a unresponsive state  ?
>>>
>>> Thanks
>>>
>>> Agateaaa
>>>
>>>
>>> Example Connection reset error:
>>> ======================>>>
>>> org.apache.thrift.transport.TTransportException:
>>> java.net.SocketException:
>>> Connection reset
>>> at
>>>
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>>  at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>> at
>>>
>>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>  at
>>>
>>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>> at
>>>
>>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>  at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>>> at
>>>
>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:2136)
>>>  at
>>>
>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:2122)
>>> at
>>>
>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openStore(HiveMetaStoreClient.java:286)
>>>  at
>>>
>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:197)
+
Nitin Pawar 2013-07-30, 07:49
+
agateaaa 2013-07-31, 16:48
+
agateaaa 2013-08-29, 17:57
+
agateaaa 2013-08-29, 18:22
+
Ashutosh Chauhan 2013-08-29, 18:53
+
agateaaa 2013-08-29, 21:39
+
agateaaa 2013-08-29, 21:39
+
agateaaa 2013-07-30, 00:21