Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: I am running MapReduce on a 30G data on 1master/2 slave, but failed.


Copy link to this message
-
Re: Re: I am running MapReduce on a 30G data on 1master/2 slave, but failed.
Thanks. I will read that document. I am new on this. So i attach the log
here for help. I can't understand this information.

These log from tasktrack log file:

2013-01-15 08:00:44,731 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0 0.66737837% reduce > reduce
2013-01-15 08:00:47,757 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0 0.66737837% reduce > reduce
2013-01-15 08:10:49,083 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201301150318_0001_r_000000_0: Task
attempt_201301150318_0001_r_000000_0 failed to report status for 601
seconds. Killing!
2013-01-15 08:10:49,092 INFO org.apache.hadoop.mapred.TaskTracker: Process
Thread Dump: lost task
24 active threads
Thread 219 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 218 (JVM Runner jvm_201301150318_0001_r_-641939786 spawned.):
  State: WAITING
  Blocked count: 1
  Waited count: 2
  Waiting on java.lang.UNIXProcess@1195929
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
    org.apache.hadoop.util.Shell.runCommand(Shell.java:244)
    org.apache.hadoop.util.Shell.run(Shell.java:182)

org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)

org.apache.hadoop.mapred.DefaultTaskController.launchTask(DefaultTaskController.java:131)

org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:497)

org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:471)
Thread 214 (Thread-135):
  State: WAITING
  Blocked count: 2
  Waited count: 3
  Waiting on java.lang.Object@1f35647
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskRunner.launchJvmAndWait(TaskRunner.java:296)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:251)
Thread 38 (IPC Client (47) connection to master/10.120.253.32:9001 from
hadoop):
  State: TIMED_WAITING
  Blocked count: 11210
  Waited count: 11199
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 10 (taskCleanup):
  State: WAITING
  Blocked count: 4
  Waited count: 7
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@104109e
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:422)
    java.lang.Thread.run(Thread.java:662)
Thread 13 (Thread-5):
  State: WAITING
  Blocked count: 475
  Waited count: 646
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61bb9b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.monitor(UserLogManager.java:131)

org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager$1.run(UserLogManager.java:66)
Thread 14 (Thread-6):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 5
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.mapred.UserLogCleaner.run(UserLogCleaner.java:93)
Thread 37 (Timer-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 618
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 36 (23978087@qtp-7433399-0 - Acceptor0
SelectChannelConnector@0.0.0.0:50060):
  State: RUNNABLE
  Blocked count: 29509
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)

org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)

org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)

org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 35 (TaskLauncher for REDUCE tasks):
  State: WAITING
  Blocked count: 1
  Waited count: 2
  Waiting on java.util.LinkedList@ca1038
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2265)
Thread 34 (TaskLauncher for MAP tasks):
  State: WAITING
  Blocked count: 170
  Waited count: 171
  Waiting on java.util.LinkedList@1ddec9e
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)

org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2265)
Thread 27 (Map-events fetcher for all reduce tasks on
tracker_ip-10-87-145-204.ec2.internal:localhost/127.0.0.1:60497):
  State: TIMED_WAITING
  Blocked count: 16184
  Waited count: 17896
  Stack:
    java.lang.Object.wait(Native Method)

org.apache.hadoop.mapred.TaskTracker$MapEventsFetcherThread.run(TaskTracker.java:975)
T
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB