Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Job stuck in running state on Hadoop 2.2.0


+
Silvina Caíno Lores 2013-12-11, 09:01
+
Silvina Caíno Lores 2013-12-11, 09:32
+
Silvina Caíno Lores 2013-12-11, 09:59
Copy link to this message
-
Re: Job stuck in running state on Hadoop 2.2.0
Adam Kawa 2013-12-11, 19:24
I am glad that I could help.

In our case, we followed mostly the configuration from here:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html
(changing
it a bit to adapt to our requirements e.g. today we run 2GB containers
instead of 3-4GB, but it might change in the future). Make also sure that
memorry allocated in mapreduce.map.java.opts is smaller than
mapreduce.map.memory.mb (the same for reduce tasks).
2013/12/11 Silvina Caíno Lores <[EMAIL PROTECTED]>

> I checked yarn-site.xml configuration and I tried to run the program
> without the memory configurations I found somewhere and assumed that would
> work (yarn.nodemanager.resource.memory-mb=2200 and
> yarn.scheduler.minimum-allocation-mb=500) following Adam's advice and the
> example worked beautifully :D Thanks a lot Adam for your suggestion!
>
> To prevent future disasters, may you recommend a configuration guide or
> give some hints in proper resource management?
>
> Thank you once more!
>
>
>
> On 11 December 2013 10:32, Silvina Caíno Lores <[EMAIL PROTECTED]>wrote:
>
>> OK that was indeed a classpath issue, which I solved by directly
>> exporting the output of hadoop classpath (ie. the list of neeed jars, see
>> this <http://doc.mapr.com/display/MapR/hadoop+classpath>) into
>> HADOOP_CLASSPATH in hadoop-env.sh and yarn-env.sh
>>
>> With this fixed, the stuck issue came back so I will study Adam's
>> suggestion
>>
>>
>> On 11 December 2013 10:01, Silvina Caíno Lores <[EMAIL PROTECTED]>wrote:
>>
>>> Actually now it seems to be running (or at least attempting to run) but
>>> I get further errors:
>>>
>>> hadoop jar
>>> ~/hadoop-2.2.0-maven/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar
>>> pi 1 100
>>>
>>> INFO mapreduce.Job: Job job_1386751964857_0001 failed with state FAILED
>>> due to: Application application_1386751964857_0001 failed 2 times due to AM
>>> Container for appattempt_1386751964857_0001_000002 exited with exitCode: 1
>>> due to: Exception from container-launch:
>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:504)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:417)
>>> at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:636)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>>
>>>
>>> I guess it seems some sort of classpath issue because of this log:
>>>
>>> /scratch/HDFS-scaino-2/logs/application_1386751964857_0001/container_1386751964857_0001_01_000001$
>>> cat stderr
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/hadoop/service/CompositeService
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)