Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - rack awareness in hadoop


+
Kishore Yellamraju 2013-04-20, 23:00
+
Aaron Eng 2013-04-20, 23:19
Copy link to this message
-
Re: rack awareness in hadoop
Marcos Luis Ortiz Valmase... 2013-04-21, 00:07
Like, Aaron say, this problem is related the Linux memory manager.
You can tune it using the vm.overcommit_memory=1.
Before to do any change, read all resources first:
http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
http://lwn.net/Articles/317814/
http://www.tldp.org/LDP/tlk/mm/memory.html

To learn more about how to tune kernel variables for Hadoop applications.
Read these links too:
First, the amazing Hadoop Operations´s book from Eric:
http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120

Hadoop Performance Tuning Guide from AMD:
http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf

Intel® Distribution for Apache Hadoop*  Software: Optimization and Tuning
Guide:
http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf

Best wishes.

2013/4/20 Aaron Eng <[EMAIL PROTECTED]>

> The problem is probably not related to the JVM memory so much as the Linux
> memory manager.  The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> which would imply this is happening when trying to create a new process.
>  The initial malloc for the new process space is being denied by the memory
> manager.  There could be many reasons why this happens, though the most
> likely is your overcommit settings and swap space.  I'd suggest reading
> through these details:
>
> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>
> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
> [EMAIL PROTECTED]> wrote:
>
>> All,
>>
>> I have posted this question to CDH ML ,  but i guess i can post it here
>> because its a general hadoop question.
>>
>> When the NN or JT gets the rack info, i guess it stores the info in
>> memory. can i ask you where in the JVM memory it will store the results (
>> perm gen ?) ? .  I am getting "cannot allocate memory on NN and JT " and
>> they have more than enough memory. when i looked at JVM usage stats i can
>> see it doesnt have enough perm free space.so if its storing the values in
>> perm gen  then there is a chance of this memory issues.
>>
>>
>> Thanks in advance !!!
>>
>>
>> exception that i see in logs :
>>
>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
>> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
>> Cannot allocate memory
>>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>         at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>>         at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>>         at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>>         at
>> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>>         at
>> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>>         at
>> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>>         at
>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>>         at
>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>>         at
>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>         at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>> allocate memory
>>         at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
Marcos Ortiz Valmaseda,
*Data-Driven Product Manager* at PDVSA
*Blog*: http://dataddict.wordpress.com/
*LinkedIn: *http://www.linkedin.com/in/marcosluis2186
*Twitter*: @marcosluis2186 <http://twitter.com/marcosluis2186>
+
Mohit Anchlia 2013-04-21, 00:19