-
Why does "HiveException: Unable to fetch table XYZ" happen?Jakub Holy 2013-02-08, 12:04
Hi everybody, I hope you'll be able to help me understand why our Hive
server occasionally starts throwing "HiveException: Unable to fetch table
<same table name>" and subsequently noe JDBC calls get through.
I have observed this mostly after having started the server so I guess
there could be some timing issue in Hive initialization. Most of the time
Hive works all right so there no defect in our configuration/infrastructure
per se.
Any idea how this could have been happening and how to prevent/fix it? We
use Hive 0.9.0 and connect to it via JDBC over Thrift. When I restart Hive,
the problem disappears.
Thanks a lot!
Here is the full log with some context around:
2013-02-08 09:28:43,943 WARN mapred.JobClient
(JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
2013-02-08 09:28:53,399 ERROR ql.Driver (SessionState.java:printError(400))
- FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2013-02-08 09:29:51,997 ERROR parse.SemanticAnalyzer
(SemanticAnalyzer.java:getMetaData(1128)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
playerevents
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:896)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:830)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:629)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:617)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jdo.JDOException: Attempt to instantiate insert request
gave an error
NestedThrowables:
java.lang.reflect.InvocationTargetException
at
org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:434)
at
org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:539)
at
org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
at
org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:800)
at
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:733)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
at $Proxy0.getTable(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:974)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:888)
... 16 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:588)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:324)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.getFetchRequest(RDBMSPersistenceHandler.java:267)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:239)
at
org.datanucleus.jdo.state.JDOStat
server occasionally starts throwing "HiveException: Unable to fetch table
<same table name>" and subsequently noe JDBC calls get through.
I have observed this mostly after having started the server so I guess
there could be some timing issue in Hive initialization. Most of the time
Hive works all right so there no defect in our configuration/infrastructure
per se.
Any idea how this could have been happening and how to prevent/fix it? We
use Hive 0.9.0 and connect to it via JDBC over Thrift. When I restart Hive,
the problem disappears.
Thanks a lot!
Here is the full log with some context around:
2013-02-08 09:28:43,943 WARN mapred.JobClient
(JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
2013-02-08 09:28:53,399 ERROR ql.Driver (SessionState.java:printError(400))
- FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2013-02-08 09:29:51,997 ERROR parse.SemanticAnalyzer
(SemanticAnalyzer.java:getMetaData(1128)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
playerevents
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:896)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:830)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:629)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:617)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jdo.JDOException: Attempt to instantiate insert request
gave an error
NestedThrowables:
java.lang.reflect.InvocationTargetException
at
org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:434)
at
org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:539)
at
org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
at
org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:800)
at
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:733)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
at $Proxy0.getTable(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:974)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:888)
... 16 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:588)
at
org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:324)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.getFetchRequest(RDBMSPersistenceHandler.java:267)
at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:239)
at
org.datanucleus.jdo.state.JDOStat
