Can you share the hive QL you are using for joins?
Bejoy K S
From: john smith <[EMAIL PROTECTED]>
Date: Mon, 19 Sep 2011 19:02:02
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Out of heap space errors on TTs
Thanks for the inputs...
Can I reduce the io.sort.mb ? (owing to the fact that I have less ram size ,
My conf files doesn't have an entry mapred.child.java.opts .. So I guess its
taking a default value of 200MB.
Also how to decide the number of tasks per TT ? I have 4 cores per node and
2GB of total memory . So how many per node maximum tasks should I set?
On Mon, Sep 19, 2011 at 6:28 PM, Uma Maheswara Rao G 72686 <
[EMAIL PROTECTED]> wrote:
> You need configure heap size for child tasks using below proprty.
> "mapred.child.java.opts" in mapred-site.xml
> by default it will be 200mb. But your io.sort.mb(300) is more than that.
> So, configure more heap space for child tasks.
> ----- Original Message -----
> From: john smith <[EMAIL PROTECTED]>
> Date: Monday, September 19, 2011 6:14 pm
> Subject: Out of heap space errors on TTs
> To: [EMAIL PROTECTED]
> > Hey guys,
> > I am running hive and I am trying to join two tables (2.2GB and
> > 136MB) on a
> > cluster of 9 nodes (replication = 3)
> > Hadoop version - 0.20.2
> > Each data node memory - 2GB
> > HADOOP_HEAPSIZE - 1000MB
> > other heap settings are defaults. My hive launches 40 Maptasks and
> > everytask failed with the same error
> > 2011-09-19 18:37:17,110 INFO org.apache.hadoop.mapred.MapTask:
> > io.sort.mb = 300
> > 2011-09-19 18:37:17,223 FATAL org.apache.hadoop.mapred.TaskTracker:
> > Error running child : java.lang.OutOfMemoryError: Java heap space
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Looks like I need to tweak some of the heap settings for TTs to handle
> > the memory efficiently. I am unable to understand which variables to
> > modify (there are too many related to heap sizes).
> > Any specific things I must look at?
> > Thanks,
> > jS