I'm running CDH 4 on  a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back

Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark

I checked the log files and the only thing that it does output is java heap space error. Nothing more.

Any help would be appreciated.

Thanks
Subash

Regards, Subash.
Can you share more information about your YARN cluster?

----- Mensaje original -----
De: Subash D'Souza <[EMAIL PROTECTED]>
Para: [EMAIL PROTECTED]
Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
Asunto: Java heap space error

I'm running CDH 4 on  a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back

Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark

I checked the log files and the only thing that it does output is java heap space error. Nothing more.

Any help would be appreciated.

Thanks
Subash

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

Try upping the child to 1.5GB or more.

On Oct 21, 2012, at 8:18 AM, Subash D'Souza <[EMAIL PROTECTED]> wrote:

> I'm running CDH 4 on  a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>

Here's the mapped-site and yarn-site configs
Mapred-site.xml
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoop1.rad.wc.truecarcorp.com:8021</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx800m</value>
</property>
</configuration>

Yarn-site.xml

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8088</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>

<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
      $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$YARN_HOME/*,$YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/data/1/yarn/local,/home/data/2/yarn/local,/home/data/3/yarn/lo
cal</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/data/1/yarn/logs,/home/data/2/yarn/logs,/home/data/3/yarn/logs
</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/home/data/tmp</value>
</property>
 <property>
   <name>yarn.nodemanager.resource.memory-mb</name>
   <value>84000</value>
   </property>
</configuration>
On 10/21/12 7:22 AM, " Marcos Ortiz Valmaseda" <[EMAIL PROTECTED]> wrote:

>Regards, Subash.
>Can you share more information about your YARN cluster?
>
>----- Mensaje original -----
>De: Subash D'Souza <[EMAIL PROTECTED]>
>Para: [EMAIL PROTECTED]
>Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
>Asunto: Java heap space error
>
>I'm running CDH 4 on  a 4 node cluster each with 96 G of RAM. Up until
>last week the cluster was running until there was an error in the name
>node log file and I had to reformat it put the data back
>
>Now when I run hive on YARN. I keep getting a Java heap space error.

Subash,

I have been experiencing this type of an error at some point and no matter how much I played with heap size  it didn't work. What I found out at the end is I was running out of physical memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space with "$hadop fs -df
From: Subash D'Souza [mailto:[EMAIL PROTECTED]]
Sent: Sunday, October 21, 2012 9:19 AM
To: [EMAIL PROTECTED]
Subject: Java heap space error

I'm running CDH 4 on  a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back

Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark

I checked the log files and the only thing that it does output is java heap space error. Nothing more.

Any help would be appreciated.

Thanks
Subash

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

Did this job ever run successfully for you? With 200m heap size?

Seems like your maps are failing. Can you paste your settings for the following:
 - io.sort.factor
 - io.sort.mb
 - mapreduce.map.sort.spill.percent

Thanks,
+Vinod

On Oct 21, 2012, at 6:18 AM, Subash D'Souza wrote:

> I'm running CDH 4 on  a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>