Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> metastore (mysql) connection problems - hive v0.12


Copy link to this message
-
Re: metastore (mysql) connection problems - hive v0.12
Hi Stephen,

Try increasing the max connections limit in MySQL. I ran into a similar
problem and this resolved things after a Hive 0.12 upgrade. You can
increase the connection limit without a restart, but pay attention to other
settings as well so that you don't over allocate resources on your db
server.

Regards,
Rick
On Dec 17, 2013 6:12 PM, "Stephen Sprague" <[EMAIL PROTECTED]> wrote:

> hi guys,
> I'm pretty much at my wits end on this one. i'm using hive v0.12 against a
> mysql metastore.   the error manifests itself (mostly randomly which is the
> problem) as this error in the client:
>
> 2013-12-17 16:54:03,158 ERROR metastore.RetryingRawStore
> (RetryingRawStore.java:invoke(146)) - JDO datastore error. Retrying
> metastore command after 1000 ms (attempt 1 of 1)
> 2013-12-17 16:54:04,345 ERROR metastore.RetryingHMSHandler
> (RetryingHMSHandler.java:invoke(141)) -
> MetaException(message:java.lang.RuntimeException: commitTransaction was
> called but openTransactionCalls = 0. This probably indicates that there are
> unbalanced calls to openTransaction/commitTransaction)
>
>
> but i believe this to be just a symptom of something happening further
> upstream.
>
>
> examining the hive.log file for this useer (under /tmp/<user>/hive.log) i
> find this hideous traceback with "bonecp" being the focus of attention.
> There's nothing wrong with the mysql server metastore especially given it
> works sometimes.  to me i think it has something to do with the connection
> pooling logic but that's only speculation.
>
> before i surrender and rollback to a previous version i ask:  Has anyone
> encountered this before?  Or are there any nuggets in the traceback worthy
> of insight?
>
> thanks,
> Stephen.
>
>
> 2013-12-17 16:54:03,142 ERROR bonecp.ConnectionHandle
> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
> Killing off all remaining connections in the connection pool. SQL State > 08S01
> 2013-12-17 16:54:03,144 ERROR bonecp.ConnectionHandle
> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
> Killing off all remaining connections in the connection pool. SQL State > 08007
> 2013-12-17 16:54:03,146 ERROR bonecp.ConnectionHandle
> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
> Killing off all remaining connections in the connection pool. SQL State > 08007
> 2013-12-17 16:54:03,147 ERROR bonecp.ConnectionHandle
> (ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
> Killing off all remaining connections in the connection pool. SQL State > 08007
> 2013-12-17 16:54:03,148 ERROR bonecp.BoneCP
> (BoneCP.java:terminateAllConnections(199)) - Error in attempting to close
> connection
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> Communications link failure during rollback(). Transaction resolution
> unknown.
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>         at com.mysql.jdbc.Util.getInstance(Util.java:386)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>         at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4730)
>         at
> com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4325)
>         at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557)
>         at
> com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:396)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB