I have been experiencing this type of an error at some point and no matter how much I played with heap size it didn't work. What I found out at the end is I was running out of physical memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space with "$hadop fs -df
From: Subash D'Souza [mailto:[EMAIL PROTECTED]]
Sent: Sunday, October 21, 2012 9:19 AM
To: [EMAIL PROTECTED]
Subject: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel