Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Job stuck in running state on Hadoop 2.2.0


Copy link to this message
-
Re: Job stuck in running state on Hadoop 2.2.0
I am glad that I could help.

In our case, we followed mostly the configuration from here:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html
(changing
it a bit to adapt to our requirements e.g. today we run 2GB containers
instead of 3-4GB, but it might change in the future). Make also sure that
memorry allocated in mapreduce.map.java.opts is smaller than
mapreduce.map.memory.mb (the same for reduce tasks).
2013/12/11 Silvina Caíno Lores <[EMAIL PROTECTED]>

> I checked yarn-site.xml configuration and I tried to run the program
> without the memory configurations I found somewhere and assumed that would
> work (yarn.nodemanager.resource.memory-mb=2200 and
> yarn.scheduler.minimum-allocation-mb=500) following Adam's advice and the
> example worked beautifully :D Thanks a lot Adam for your suggestion!
>
> To prevent future disasters, may you recommend a configuration guide or
> give some hints in proper resource management?
>
> Thank you once more!
>
>
>
> On 11 December 2013 10:32, Silvina Caíno Lores <[EMAIL PROTECTED]>wrote:
>
>> OK that was indeed a classpath issue, which I solved by directly
>> exporting the output of hadoop classpath (ie. the list of neeed jars, see
>> this <http://doc.mapr.com/display/MapR/hadoop+classpath>) into
>> HADOOP_CLASSPATH in hadoop-env.sh and yarn-env.sh
>>
>> With this fixed, the stuck issue came back so I will study Adam's
>> suggestion
>>
>>
>> On 11 December 2013 10:01, Silvina Caíno Lores <[EMAIL PROTECTED]>wrote:
>>
>>> Actually now it seems to be running (or at least attempting to run) but
>>> I get further errors:
>>>
>>> hadoop jar
>>> ~/hadoop-2.2.0-maven/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar
>>> pi 1 100
>>>
>>> INFO mapreduce.Job: Job job_1386751964857_0001 failed with state FAILED
>>> due to: Application application_1386751964857_0001 failed 2 times due to AM
>>> Container for appattempt_1386751964857_0001_000002 exited with exitCode: 1
>>> due to: Exception from container-launch:
>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:504)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:417)
>>> at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:636)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>>
>>>
>>> I guess it seems some sort of classpath issue because of this log:
>>>
>>> /scratch/HDFS-scaino-2/logs/application_1386751964857_0001/container_1386751964857_0001_01_000001$
>>> cat stderr
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/hadoop/service/CompositeService
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB