Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> metastore (mysql) connection problems - hive v0.12


Copy link to this message
-
Re: metastore (mysql) connection problems - hive v0.12
well. its back again. so while max_connections definitely should be
increased for me anyway it isn't the root cause. its currently set to 500
which is plenty.  turning on the debugging log level to '2' on the
mysql/percona server its now reporting drop packets.  doesn't say why but
we're still digging.   i dunno. this datanucleus/bonecp component seems
like an area of interest for the mysql metastore.

i'll let the group know if i find anything more useful.
On Wed, Dec 18, 2013 at 2:15 PM, Stephen Sprague <[EMAIL PROTECTED]> wrote:

> Thanks for the response and suggestion Rick.
>
> That certainly seems like a plausible solution given that we are using the
> mysql server as a metastore for more than a few dev and prod hive
> instances.   I think also with hive v0.12 the connection pooling middleware
> is different (BoneCP) so who knows what that brought into the mix.  And...
> given the randomness of the error it sure smells like a runtime-y kinda
> thing.
>
> so far it looks like its working!
>
> thanks,
> stephen.
>
>
> On Tue, Dec 17, 2013 at 6:10 PM, Richard Nadeau <[EMAIL PROTECTED]>wrote:
>
>> Hi Stephen,
>>
>> Try increasing the max connections limit in MySQL. I ran into a similar
>> problem and this resolved things after a Hive 0.12 upgrade. You can
>> increase the connection limit without a restart, but pay attention to other
>> settings as well so that you don't over allocate resources on your db
>> server.
>>
>> Regards,
>> Rick
>> On Dec 17, 2013 6:12 PM, "Stephen Sprague" <[EMAIL PROTECTED]> wrote:
>>
>>> hi guys,
>>> I'm pretty much at my wits end on this one. i'm using hive v0.12 against
>>> a mysql metastore.   the error manifests itself (mostly randomly which is
>>> the problem) as this error in the client:
>>>
>>> 2013-12-17 16:54:03,158 ERROR metastore.RetryingRawStore
>>> (RetryingRawStore.java:invoke(146)) - JDO datastore error. Retrying
>>> metastore command after 1000 ms (attempt 1 of 1)
>>> 2013-12-17 16:54:04,345 ERROR metastore.RetryingHMSHandler
>>> (RetryingHMSHandler.java:invoke(141)) -
>>> MetaException(message:java.lang.RuntimeException: commitTransaction was
>>> called but openTransactionCalls = 0. This probably indicates that there are
>>> unbalanced calls to openTransaction/commitTransaction)
>>>
>>>
>>> but i believe this to be just a symptom of something happening further
>>> upstream.
>>>
>>>
>>> examining the hive.log file for this useer (under /tmp/<user>/hive.log)
>>> i find this hideous traceback with "bonecp" being the focus of attention.
>>> There's nothing wrong with the mysql server metastore especially given it
>>> works sometimes.  to me i think it has something to do with the connection
>>> pooling logic but that's only speculation.
>>>
>>> before i surrender and rollback to a previous version i ask:  Has anyone
>>> encountered this before?  Or are there any nuggets in the traceback worthy
>>> of insight?
>>>
>>> thanks,
>>> Stephen.
>>>
>>>
>>> 2013-12-17 16:54:03,142 ERROR bonecp.ConnectionHandle
>>> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
>>> Killing off all remaining connections in the connection pool. SQL State >>> 08S01
>>> 2013-12-17 16:54:03,144 ERROR bonecp.ConnectionHandle
>>> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
>>> Killing off all remaining connections in the connection pool. SQL State >>> 08007
>>> 2013-12-17 16:54:03,146 ERROR bonecp.ConnectionHandle
>>> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
>>> Killing off all remaining connections in the connection pool. SQL State >>> 08007
>>> 2013-12-17 16:54:03,147 ERROR bonecp.ConnectionHandle
>>> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
>>> Killing off all remaining connections in the connection pool. SQL State >>> 08007
>>> 2013-12-17 16:54:03,148 ERROR bonecp.BoneCP
>>> (BoneCP.java:terminateAllConnections(199)) - Error in attempting to close
>>> connection
>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB