Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> RC -> ORC INSERT OVERWRITE metastore Heap Error


Copy link to this message
-
Re: RC -> ORC INSERT OVERWRITE metastore Heap Error
I'll just go ahead an answer my own question here. In my hive script I had
a ghost "set hive.optimize.ppd=false;"  That, I believe, caused my
metastore operations to be far in excess of what I was trying to do. Things
are working now. I just didn't want to ask a question, answer it (due to my
own error) and not have other be able to learn from my mistakes. Sorry
group!

John
On Sat, Aug 10, 2013 at 8:18 AM, John Omernik <[EMAIL PROTECTED]> wrote:

> I am doing some testing going from table_rc to table_orc . The
> table/partition structure is the same, and there is a two level partition
> day= then source>
> I am doing a single day (including all 10 or so sources in the day). This
> worked just fine in one environment, but now, I am getting strange errors
> before things even get to the data (or so it seams). The error seems to be
> related to the metastore, but on the table_rc table rather than the orc.  I
> guess I am curious why this error is happening. This is on the mapr
> distribution of Hadoop. But it seems more related to metastore operations
> than hadoop filesystem operations.
>
>
> INSERT OVERWRITE TABLE table_orc PARTITION (day, source)
> SELECT * from table_rc where day = '2013-08-04'
> 13/08/10 08:44:41 INFO parse.ParseDriver: Parse Completed
> 13/08/10 08:44:41 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
> 13/08/10 08:44:41 INFO parse.SemanticAnalyzer: Completed phase 1 of
> Semantic Analysis
> 13/08/10 08:44:41 INFO parse.SemanticAnalyzer: Get metadata for source
> tables
> 13/08/10 08:44:41 INFO metastore.HiveMetaStore: 0: get_table :
> db=hive_flow tbl=pcaps
> 13/08/10 08:44:41 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_table
> : db=hive_flow tbl=table_rc
> 13/08/10 08:44:41 INFO parse.SemanticAnalyzer: Get metadata for subqueries
> 13/08/10 08:44:41 INFO parse.SemanticAnalyzer: Get metadata for
> destination tables
> 13/08/10 08:44:41 INFO metastore.HiveMetaStore: 0: get_table :
> db=hive_flow tbl=table_orc
> 13/08/10 08:44:41 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_table
> : db=hive_flow tbl=table_orc
> 13/08/10 08:44:41 INFO parse.SemanticAnalyzer: Completed getting MetaData
> in Semantic Analysis
> 13/08/10 08:44:41 INFO metastore.HiveMetaStore: 0:
> get_partitions_with_auth : db=hive_flow tbl=table_rc
> 13/08/10 08:44:41 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_partitions_with_auth
> : db=hive_flow tbl=table_rc
> 13/08/10 08:52:12 ERROR metastore.RetryingHMSHandler:
> java.lang.OutOfMemoryError: Java heap space
> at com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2098)
>  at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:459)
> at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2076)
>  at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1451)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1787)
>  at com.mysql.jdbc.Connection.execSQL(Connection.java:3283)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1332)
>  at
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1467)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
>  at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> at
> org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:457)
>  at
> org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java:248)
> at
> org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987)
>  at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258)
> at org.datanucleus.sco.backed.Map.keySet(Map.java:509)
>  at
> org.datanucleus.store.fieldmanager.LoadFieldManager.internalFetchObjectField(LoadFieldManager.java:118)
> at
> org.datanucleus.store.fieldmanager.AbstractFetchFieldManager.fetchObjectField(AbstractFetchFieldManager.java:114)
>  at
> org.datanucleus.state.AbstractStateManager.replacingObjectField(AbstractStateManager.java:1183)