Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: I am running MapReduce on a 30G data on 1master/2 slave, but failed.


Copy link to this message
-
Re: Re: I am running MapReduce on a 30G data on 1master/2 slave, but failed.
yaotian 2013-01-16, 03:23
Thanks. I will read that document. I am new on this. So i attach the log
here for help. I can't understand this information.

These log from tasktrack log file:

2013-01-15 08:00:44,731 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0 0.66737837% reduce > reduce
2013-01-15 08:00:47,757 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0 0.66737837% reduce > reduce
2013-01-15 08:10:49,083 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0: Task
attempt_201301150318_0001_r_000000_0 failed to report status for 601
seconds. Killing!
2013-01-15 08:10:49,092 INFO org.apache.hadoop.mapred.TaskTracker: Process
Thread Dump: lost task
24 active threads
Thread 219 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 218 (JVM Runner jvm_201301150318_0001_r_-641939786 spawned.):
  State: WAITING
  Blocked count: 1
  Waited count: 2
  Waiting on java.lang.UNIXProcess@1195929
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
    org.apache.hadoop.util.Shell.runCommand(Shell.java:244)
    org.apache.hadoop.util.Shell.run(Shell.java:182)

org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)

org.apache.hadoop.mapred.DefaultTaskController.launchTask(DefaultTaskController.java:131)

org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:497)

org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:471)
Thread 214 (Thread-135):
  State: WAITING
  Blocked count: 2
  Waited count: 3
  Waiting on java.lang.Object@1f35647
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskRunner.launchJvmAndWait(TaskRunner.java:296)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:251)
Thread 38 (IPC Client (47) connection to master/10.120.253.32:9001 from
hadoop):
  State: TIMED_WAITING
  Blocked count: 11210
  Waited count: 11199
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 10 (taskCleanup):
  State: WAITING
  Blocked count: 4
  Waited count: 7
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@104109e
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:422)
    java.lang.Thread.run(Thread.java:662)
Thread 13 (Thread-5):
  State: WAITING
  Blocked count: 475
  Waited count: 646
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61bb9b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.monitor(UserLogManager.java:131)

org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager$1.run(UserLogManager.java:66)
Thread 14 (Thread-6):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 5
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.mapred.UserLogCleaner.run(UserLogCleaner.java:93)
Thread 37 (Timer-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 618
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 36 (23978087@qtp-7433399-0 - Acceptor0
SelectChannelConnector@0.0.0.0:50060):
  State: RUNNABLE
  Blocked count: 29509
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)

org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)

org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)

org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 35 (TaskLauncher for REDUCE tasks):
  State: WAITING
  Blocked count: 1
  Waited count: 2
  Waiting on java.util.LinkedList@ca1038
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2265)
Thread 34 (TaskLauncher for MAP tasks):
  State: WAITING
  Blocked count: 170
  Waited count: 171
  Waiting on java.util.LinkedList@1ddec9e
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2265)
Thread 27 (Map-events fetcher for all reduce tasks on
tracker_ip-10-87-145-204.ec2.internal:localhost/127.0.0.1:60497):
  State: TIMED_WAITING
  Blocked count: 16184
  Waited count: 17896
  Stack:
    java.lang.Object.wait(Native Method)

org.apache.hadoop.mapred.TaskTracker$MapEventsFetcherThread.run(TaskTracker.java:975)
T