Hi everybody, I hope you'll be able to help me understand why our Hive
server occasionally starts throwing "HiveException: Unable to fetch table
<same table name>" and subsequently noe JDBC calls get through.

I have observed this mostly after having started the server so I guess
there could be some timing issue in Hive initialization. Most of the time
Hive works all right so there no defect in our configuration/infrastructure
per se.

Any idea how this could have been happening and how to prevent/fix it? We
use Hive 0.9.0 and connect to it via JDBC over Thrift. When I restart Hive,
the problem disappears.

Thanks a lot!

Here is the full log with some context around:

2013-02-08 09:28:43,943 WARN  mapred.JobClient
(JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
2013-02-08 09:28:53,399 ERROR ql.Driver (SessionState.java:printError(400))
- FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2013-02-08 09:29:51,997 ERROR parse.SemanticAnalyzer
(SemanticAnalyzer.java:getMetaData(1128)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
playerevents
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:896)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:830)
        at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954)
        at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524)
        at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
        at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
        at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:629)
        at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:617)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jdo.JDOException: Attempt to instantiate insert request
gave an error
NestedThrowables:
java.lang.reflect.InvocationTargetException
        at
org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:434)
        at
org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:539)
        at
org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
        at
org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
        at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:800)
        at
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:733)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
        at $Proxy0.getTable(Unknown Source)
        at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:974)
        at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:888)
        ... 16 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:588)
        at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:324)
        at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.getFetchRequest(RDBMSPersistenceHandler.java:267)
        at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:239)
        at
org.datanucleus.jdo.state.JDOStat

In short, hiveserver has leaks, the upcoming hiveserver2 (and the
changes made in hive trunk due to it should prevent many of these. We
run our hiveserver's behind a load balancer (4 of them) basically this
increases the mean time to failure due to leaks. It would be good if
hive-server had some jmx or some method to determine # of connected
clients and then an admin can script a restart in a dead time.

IF you are using zk locking turn that off, zklocking+hiveserver leaks
much more then just plan hiveserver.

On Fri, Feb 8, 2013 at 7:04 AM, Jakub Holy <[EMAIL PROTECTED]> wrote:
> Hi everybody, I hope you'll be able to help me understand why our Hive
> server occasionally starts throwing "HiveException: Unable to fetch table
> <same table name>" and subsequently noe JDBC calls get through.
>
> I have observed this mostly after having started the server so I guess there
> could be some timing issue in Hive initialization. Most of the time Hive
> works all right so there no defect in our configuration/infrastructure per
> se.
>
> Any idea how this could have been happening and how to prevent/fix it? We
> use Hive 0.9.0 and connect to it via JDBC over Thrift. When I restart Hive,
> the problem disappears.
>
> Thanks a lot!
>
> Here is the full log with some context around:
>
> 2013-02-08 09:28:43,943 WARN  mapred.JobClient
> (JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 2013-02-08 09:28:53,399 ERROR ql.Driver (SessionState.java:printError(400))
> - FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> 2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.resources" but it cannot be resolved.
> 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.core.runtime" but it cannot be resolved.
> 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
> "org.eclipse.text" but it cannot be resolved.
> 2013-02-08 09:29:51,997 ERROR parse.SemanticAnalyzer
> (SemanticAnalyzer.java:getMetaData(1128)) -
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
> playerevents
>         at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:896)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:830)
>         at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954)
>         at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524)
>         at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
>         at
> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
>         at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:629)
>         at
> org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:617)
>         at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)