|
|
-
Fwd: YARN NM containers were killedYouPeng Yang 2013-01-31, 04:28
Hi
I have posted my question for a day,please can somebody help me to figure out what the problem is. Thank you. regards YouPeng Yang ---------- Forwarded message ---------- From: YouPeng Yang <[EMAIL PROTECTED]> Date: 2013/1/30 Subject: YARN NM containers were killed To: [EMAIL PROTECTED] i've tested the hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar on my hadoop environment ( 1 RM - Hadoop01 and 3 NM --Hadoop02,Hadoop03,Hadoop04 OS:CDH4.1.2 rhel5.5): ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar wordcount 1/input output when i checked the log .i was confused by the plz: my hadoop creates 2 containers in Hadoop02,1 container in Hadoop03 ,however 0 container Hadoop04. the result of the containers processing: Hadoop02: * container_1359422495723_0001_01_000001 (its state changes as follows:NEW --> LOCALIZING --> LOCALIZED --> RUNNING --> KILLING --> EXITED_WITH_SUCCESS) the log indates that: NodeStatusUpdaterImpl: Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1359422495723, }, attemptId: 1, }, id: 1, }, state: C_RUNNING, diagnostics: "", exit_status: -1000, ContainerLaunch: Container container_1359422495723_0001_01_000001 succeeded Container: Container container_1359422495723_0001_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS ContainerLaunch: Cleaning up container container_1359422495723_0001_01_000001 NMAuditLogger: USER=hadoop OPERATION=Container Finished - Succeeded TARGET=ContainerImpl RESULT=SUCCESSAPPID=application_1359422495723_0001 CONTAINERID=container_1359422495723_0001_01_000001 * container_1359422495723_0001_01_000003 (its state changes as follows:NEW --> LOCALIZING --> LOCALIZED --> RUNNING --> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE) the log indates that: NodeStatusUpdaterImpl: Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1359422495723, }, attemptId: 1, }, id: 3, }, state: C_RUNNING, diagnostics: "Container killed by the ApplicationMaster.\n", exit_status: -1000, DefaultContainerExecutor: Exit code from task is : 137 NMAuditLogger: USER=hadoop OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1359422495723_0001 CONTAINERID=container_1359422495723_0001_01_000003 Hadoop03: * container_1359422495723_0001_01_000002 (its state changes as follows:NEW --> LOCALIZING --> LOCALIZED --> RUNNING --> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE) NodeStatusUpdaterImpl: Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1359422495723, }, attemptId: 1, }, id: 2, }, state: C_RUNNING, diagnostics: "Container killed by the ApplicationMaster.\n", exit_status: -1000, DefaultContainerExecutor: Exit code from task is : 143 NMAuditLogger: USER=hadoop OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1359422495723_0001 CONTAINERID=container_1359422495723_0001_01_000002 My questions: 1. Why were 2 containers created in Hadoop02,however Hadoop04 got nothing.is it normal ? 2. What is the principle that guides containers to be created. 3. Why were the two containers (the container_*_000003 and the container_*_000002) killed, while the container_*_000001 succeeded. is it normal? logs of Hadoop01 as follows: 2013-01-29 09:23:48,904 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 1 2013-01-29 09:23:50,201 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 1 submitted by user hadoop 2013-01-29 09:23:50,204 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop IP=10.167.14.221 OPERATION=Submit Application Request TARGET=ClientRMServiceRESULT=SUCCESS APPID=application_1359422495723_0001 2013-01-29 09:23:50,221 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1359422495723_0001 State change from NEW to SUBMITTED 2013-01-29 09:23:50,221 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering appattempt_1359422495723_0001_000001 2013-01-29 09:23:50,222 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1359422495723_0001_000001 State change from NEW to SUBMITTED 2013-01-29 09:23:50,242 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler: Application Submission: application_1359422495723_0001 from hadoop, currently active: 1 2013-01-29 09:23:50,250 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1359422495723_0001_000001 State change from SUBMITTED to SCHEDULED 2013-01-29 09:23:50,250 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1359422495723_0001 State change from SUBMITTED to ACCEPTED 2013-01-29 09:23:50,581 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1359422495723_0001_01_000001 Container Transitioned from NEW to ALLOCATED 2013-01-29 09:23:50,581 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1359422495723_0001 CONTAINERID=container_1359422495723_0001_01_000001 2013-01-29 09:23:50,581 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode: Assigned container container_1359422495723_0001_01_000001 of capacity memory: 1536 on host Hadoop02:39876, which currently has 1 containers, memory: 1536 used and memory: 6656 available 2013-01-29 09:23:50,582 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1359422495723_0001_01_000001 Container Transitioned from ALLOCATED to ACQUIRED 2013-01-29 09:23:50,583 INFO org.apache.hadoop.yarn.server.resource |