Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hive running out of memory


Copy link to this message
-
Re: Hive running out of memory
Igor Tatarinov 2011-06-21, 21:31
Yes, that's probably it. I found a related JIRA:
https://issues.apache.org/jira/browse/HIVE-1316

doesn't look like the EMR installation has this fix. I am going to increase
the heap size and see if that helps.

On Tue, Jun 21, 2011 at 1:52 PM, Steven Wong <[EMAIL PROTECTED]> wrote:

> Is the OOM in the Hive client? If so, you should try increasing its max
> heap size by setting the env var HADOOP_HEAPSIZE. One place to set it in is
> hive-env.sh; see /home/hadoop/.versions/hive-0.7/conf/hive-env.sh.template
> for more info.****
>
> ** **
>
> ** **
>
> *From:* Igor Tatarinov [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, June 21, 2011 12:19 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* Hive running out of memory****
>
> ** **
>
> I have a table with 3 levels of partitioning and about 10,000 files (one
> file at every 'leaf'). I am using EMR and the table is stored in S3.****
>
> For some reason, Hive can't even start running a simple query that creates
> a local copy of a subset of the big table.****
>
> ** **
>
> Does this look like an EMR-specific issue or is there something I could do?
> ****
>
> I am thinking about copying al of the data into HDFS first.****
>
> ** **
>
> Number of reduce tasks is set to 0 since there's no reduce operator****
>
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded****
>
>             at
> java.util.LinkedHashMap.newKeyIterator(LinkedHashMap.java:396)****
>
>             at java.util.HashMap$KeySet.iterator(HashMap.java:874)****
>
>             at
> java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:516)
> ****
>
>             at
> java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:190)***
> *
>
>             at
> java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:393)
> ****
>
>             at
> java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:190)***
> *
>
>             at
> java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:393)
> ****
>
>             at
> java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:100)**
> **
>
>             at java.beans.Encoder.writeObject(Encoder.java:54)****
>
>             at java.beans.XMLEncoder.writeObject(XMLEncoder.java:257)****
>
>             at java.beans.Encoder.writeExpression(Encoder.java:279)****
>
>             at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:372)*
> ***
>
>             at
> java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:97)***
> *
>
>             at java.beans.Encoder.writeObject(Encoder.java:54)****
>
>             at java.beans.XMLEncoder.writeObject(XMLEncoder.java:257)****
>
>             at java.beans.Encoder.writeExpression(Encoder.java:279)****
>
>             at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:372)*
> ***
>
>             at
> java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:212)
> ****
>
>             at
> java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:247)
> ****
>
>             at
> java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:395)
> ****
>
>             at
> java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:100)**
> **
>
>             at java.beans.Encoder.writeObject(Encoder.java:54)****
>
>             at java.beans.XMLEncoder.writeObject(XMLEncoder.java:257)****
>
>             at java.beans.Encoder.writeExpression(Encoder.java:279)****
>
>             at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:372)*
> ***
>
>             at
> java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:97)***
> *
>
>             at java.beans.Encoder.writeObject(Encoder.java:54)****
>
>             at java.beans.XMLEncoder.writeObject(XMLEncoder.java:257)****
>
>             at java.beans.Encoder.writeExpression(Encoder.java:279)****
>
>             at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:372)*