Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Java Heap space Exception


+
unmesha sreeveni 2013-11-15, 09:23
+
unmesha sreeveni 2013-11-15, 11:03
+
Rob Blah 2013-11-15, 13:16
Copy link to this message
-
Re: Java Heap space Exception
Rob Blah 2013-11-15, 13:24
MapReduce
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
"We’ll thus assign 4 GB for Map task Containers, and 8 GB for Reduce tasks
Containers."

You can try editing configuration file:
<name>mapreduce.map.memory.mb</name>
<value>?</value>

<name>mapreduce.reduce.memory.mb</name>
 <value>?</value>

Or set these in a driver.

If you have custom AM, try this:
Driver:
capability.setMemory(amMemory);
appContext.setResource(capability);

AM:
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(containerMemory);
AMRMClient.ContainerRequest request = new
AMRMClient.ContainerRequest(capability, null, null, pri);
OR AM (use java command to set heap size):
vargs.add("-Xmx" + containerMemory + "m");
2013/11/15 Rob Blah <[EMAIL PROTECTED]>

> Question have you tried giving more memory to the containers?
>
>
> 2013/11/15 unmesha sreeveni <[EMAIL PROTECTED]>
>
>> ys found
>> tried -D mapred.child.java.opts=-Xmx4096M on the command line:
>>
>>
>> On Fri, Nov 15, 2013 at 3:01 PM, unmesha sreeveni <[EMAIL PROTECTED]>wrote:
>>
>>> hadoop jar /home/my/hadoop2.jar /user/unmesha/inputdata /user/unmesha
>>> /out
>>>
>>>
>>>
>>> On Fri, Nov 15, 2013 at 2:53 PM, unmesha sreeveni <[EMAIL PROTECTED]
>>> > wrote:
>>>
>>>> When i tried to execute my program with 100 MB file i am getting
>>>> JavaHeapSpace Exception
>>>>
>>>>
>>>> >hadoop jar /home/my/hadoop2.jar /user/unmesha/inputdata
>>>> /user/inputdata/out
>>>>
>>>> How to increase the heap size through commandline
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>>
>>
>
+
unmesha sreeveni 2013-11-16, 03:08