Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - Hive Metastore Server 0.9 Connection Reset and Connection Timeout errors


+
agateaaa 2013-07-29, 17:43
+
Nitin Pawar 2013-07-29, 18:02
Copy link to this message
-
Re: Hive Metastore Server 0.9 Connection Reset and Connection Timeout errors
agateaaa 2013-07-29, 18:29
Thanks Nitin!

We have simiar setup (identical hcatalog and hive server versions) on a
another production environment and dont see any errors (its been running ok
for a few months)

Unfortunately we wont be able to move to hcat 0.5 and hive 0.11 or hive
0.10 soon.

I did see that the last time we ran into this problem doing a netstat-ntp |
grep ":10000" see that server was holding on to one socket connection in
CLOSE_WAIT state for a long time
 (hive metastore server is running on port 10000). Dont know if thats
relevant here or not

Can you suggest any hive configuration settings we can tweak or networking
tools/tips, we can use to narrow this down ?

Thanks
Agateaaa
On Mon, Jul 29, 2013 at 11:02 AM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> Is there any chance you can do a update on test environment with hcat-0.5
> and hive-0(11 or 10) and see if you can reproduce the issue?
>
> We used to see this error when there was load on hcat server or some
> network issue connecting to the server(second one was rare occurrence)
>
>
> On Mon, Jul 29, 2013 at 11:13 PM, agateaaa <[EMAIL PROTECTED]> wrote:
>
>> Hi All:
>>
>> We are running into frequent problem using HCatalog 0.4.1 (HIve Metastore
>> Server 0.9) where we get connection reset or connection timeout errors.
>>
>> The hive metastore server has been allocated enough (12G) memory.
>>
>> This is a critical problem for us and would appreciate if anyone has any
>> pointers.
>>
>> We did add a retry logic in our client, which seems to help, but I am just
>> wondering how can we narrow down to the root cause
>> of this problem. Could this be a hiccup in networking which causes the
>> hive
>> server to get into a unresponsive state  ?
>>
>> Thanks
>>
>> Agateaaa
>>
>>
>> Example Connection reset error:
>> ======================>>
>> org.apache.thrift.transport.TTransportException: java.net.SocketException:
>> Connection reset
>> at
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>  at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> at
>>
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>  at
>>
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>> at
>>
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>  at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>> at
>>
>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:2136)
>>  at
>>
>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:2122)
>> at
>>
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openStore(HiveMetaStoreClient.java:286)
>>  at
>>
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:197)
>> at
>>
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:157)
>>  at
>>
>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2092)
>> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2102)
>>  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:888)
>> at
>>
>> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTableAddParts(DDLSemanticAnalyzer.java:1817)
>>  at
>>
>> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:297)
>> at
>>
>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
>>  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
>> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
>>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
>> at
>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
>>  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
>>  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
+
agateaaa 2013-07-30, 00:22
+
Nitin Pawar 2013-07-30, 07:49
+
agateaaa 2013-07-31, 16:48
+
agateaaa 2013-08-29, 17:57
+
agateaaa 2013-08-29, 18:22
+
Ashutosh Chauhan 2013-08-29, 18:53
+
agateaaa 2013-08-29, 21:39
+
agateaaa 2013-08-29, 21:39
+
agateaaa 2013-07-30, 00:21