Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> metastore (mysql) connection problems - hive v0.12


Copy link to this message
-
Re: metastore (mysql) connection problems - hive v0.12
Hi Stephen,

Try increasing the max connections limit in MySQL. I ran into a similar
problem and this resolved things after a Hive 0.12 upgrade. You can
increase the connection limit without a restart, but pay attention to other
settings as well so that you don't over allocate resources on your db
server.

Regards,
Rick
On Dec 17, 2013 6:12 PM, "Stephen Sprague" <[EMAIL PROTECTED]> wrote:

> hi guys,
> I'm pretty much at my wits end on this one. i'm using hive v0.12 against a
> mysql metastore.   the error manifests itself (mostly randomly which is the
> problem) as this error in the client:
>
> 2013-12-17 16:54:03,158 ERROR metastore.RetryingRawStore
> (RetryingRawStore.java:invoke(146)) - JDO datastore error. Retrying
> metastore command after 1000 ms (attempt 1 of 1)
> 2013-12-17 16:54:04,345 ERROR metastore.RetryingHMSHandler
> (RetryingHMSHandler.java:invoke(141)) -
> MetaException(message:java.lang.RuntimeException: commitTransaction was
> called but openTransactionCalls = 0. This probably indicates that there are
> unbalanced calls to openTransaction/commitTransaction)
>
>
> but i believe this to be just a symptom of something happening further
> upstream.
>
>
> examining the hive.log file for this useer (under /tmp/<user>/hive.log) i
> find this hideous traceback with "bonecp" being the focus of attention.
> There's nothing wrong with the mysql server metastore especially given it
> works sometimes.  to me i think it has something to do with the connection
> pooling logic but that's only speculation.
>
> before i surrender and rollback to a previous version i ask:  Has anyone
> encountered this before?  Or are there any nuggets in the traceback worthy
> of insight?
>
> thanks,
> Stephen.
>
>
> 2013-12-17 16:54:03,142 ERROR bonecp.ConnectionHandle
> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
> Killing off all remaining connections in the connection pool. SQL State > 08S01
> 2013-12-17 16:54:03,144 ERROR bonecp.ConnectionHandle
> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
> Killing off all remaining connections in the connection pool. SQL State > 08007
> 2013-12-17 16:54:03,146 ERROR bonecp.ConnectionHandle
> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
> Killing off all remaining connections in the connection pool. SQL State > 08007
> 2013-12-17 16:54:03,147 ERROR bonecp.ConnectionHandle
> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
> Killing off all remaining connections in the connection pool. SQL State > 08007
> 2013-12-17 16:54:03,148 ERROR bonecp.BoneCP
> (BoneCP.java:terminateAllConnections(199)) - Error in attempting to close
> connection
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> Communications link failure during rollback(). Transaction resolution
> unknown.
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>         at com.mysql.jdbc.Util.getInstance(Util.java:386)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>         at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4730)
>         at
> com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4325)
>         at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557)
>         at
> com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:396)