Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive running out of memory


Copy link to this message
-
Re: Hive running out of memory
Yes, that's probably it. I found a related JIRA:
https://issues.apache.org/jira/browse/HIVE-1316

doesn't look like the EMR installation has this fix. I am going to increase
the heap size and see if that helps.

On Tue, Jun 21, 2011 at 1:52 PM, Steven Wong <[EMAIL PROTECTED]> wrote:

> Is the OOM in the Hive client? If so, you should try increasing its max
> heap size by setting the env var HADOOP_HEAPSIZE. One place to set it in is
> hive-env.sh; see /home/hadoop/.versions/hive-0.7/conf/hive-env.sh.template
> for more info.****
>
> ** **
>
> ** **
>
> *From:* Igor Tatarinov [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, June 21, 2011 12:19 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* Hive running out of memory****
>
> ** **
>
> I have a table with 3 levels of partitioning and about 10,000 files (one
> file at every 'leaf'). I am using EMR and the table is stored in S3.****
>
> For some reason, Hive can't even start running a simple query that creates
> a local copy of a subset of the big table.****
>
> ** **
>
> Does this look like an EMR-specific issue or is there something I could do?
> ****
>
> I am thinking about copying al of the data into HDFS first.****
>
> ** **
>
> Number of reduce tasks is set to 0 since there's no reduce operator****
>
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded****
>
>             at
> java.util.LinkedHashMap.newKeyIterator(LinkedHashMap.java:396)****
>
>             at java.util.HashMap$KeySet.iterator(HashMap.java:874)****
>
>             at
> java.beans.java_util_Map_PersistenceDelegate.initialize(MetaData.java:516)
> ****
>
>             at
> java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:190)***
> *
>
>             at
> java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:393)
> ****
>
>             at
> java.beans.PersistenceDelegate.initialize(PersistenceDelegate.java:190)***
> *
>
>             at
> java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:393)
> ****
>
>             at
> java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:100)**
> **
>
>             at java.beans.Encoder.writeObject(Encoder.java:54)****
>
>             at java.beans.XMLEncoder.writeObject(XMLEncoder.java:257)****
>
>             at java.beans.Encoder.writeExpression(Encoder.java:279)****
>
>             at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:372)*
> ***
>
>             at
> java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:97)***
> *
>
>             at java.beans.Encoder.writeObject(Encoder.java:54)****
>
>             at java.beans.XMLEncoder.writeObject(XMLEncoder.java:257)****
>
>             at java.beans.Encoder.writeExpression(Encoder.java:279)****
>
>             at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:372)*
> ***
>
>             at
> java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:212)
> ****
>
>             at
> java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:247)
> ****
>
>             at
> java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:395)
> ****
>
>             at
> java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:100)**
> **
>
>             at java.beans.Encoder.writeObject(Encoder.java:54)****
>
>             at java.beans.XMLEncoder.writeObject(XMLEncoder.java:257)****
>
>             at java.beans.Encoder.writeExpression(Encoder.java:279)****
>
>             at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:372)*
> ***
>
>             at
> java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:97)***
> *
>
>             at java.beans.Encoder.writeObject(Encoder.java:54)****
>
>             at java.beans.XMLEncoder.writeObject(XMLEncoder.java:257)****
>
>             at java.beans.Encoder.writeExpression(Encoder.java:279)****
>
>             at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:372)*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB