Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> YARN NM containers were killed


Copy link to this message
-
YARN NM containers were killed
i've tested the hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar on my hadoop
environment
(   1 RM - Hadoop01 and 3 NM --Hadoop02,Hadoop03,Hadoop04
  OS:CDH4.1.2 rhel5.5):
./bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar
 wordcount 1/input output

when i checked the log .i was confused by the plz:
my hadoop creates 2 containers in Hadoop02,1 container in Hadoop03 ,however
0 container Hadoop04.

the result of the containers processing:

Hadoop02:
* container_1359422495723_0001_01_000001
(its state changes as follows:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
--> KILLING --> EXITED_WITH_SUCCESS)

      the log indates that:
NodeStatusUpdaterImpl: Sending out status for container: container_id {,
app_attempt_id {, application_id {, id: 1, cluster_timestamp:
1359422495723, }, attemptId: 1, }, id: 1, }, state: C_RUNNING, diagnostics:
"", exit_status: -1000,
ContainerLaunch: Container container_1359422495723_0001_01_000001 succeeded
Container: Container container_1359422495723_0001_01_000001 transitioned
from RUNNING to EXITED_WITH_SUCCESS
ContainerLaunch: Cleaning up container
container_1359422495723_0001_01_000001
NMAuditLogger: USER=hadoop OPERATION=Container Finished - Succeeded
TARGET=ContainerImpl RESULT=SUCCESSAPPID=application_1359422495723_0001
CONTAINERID=container_1359422495723_0001_01_000001
 * container_1359422495723_0001_01_000003
(its state changes as follows:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
--> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE)
the log indates that:
NodeStatusUpdaterImpl: Sending out status for container: container_id {,
app_attempt_id {, application_id {, id: 1, cluster_timestamp:
1359422495723, }, attemptId: 1, }, id: 3, }, state: C_RUNNING, diagnostics:
"Container killed by the ApplicationMaster.\n", exit_status: -1000,
DefaultContainerExecutor: Exit code from task is : 137
NMAuditLogger: USER=hadoop OPERATION=Container Finished - Killed
TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1359422495723_0001
CONTAINERID=container_1359422495723_0001_01_000003

Hadoop03:
        * container_1359422495723_0001_01_000002
(its state changes as follows:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
--> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE)
NodeStatusUpdaterImpl: Sending out status for container: container_id {,
app_attempt_id {, application_id {, id: 1, cluster_timestamp:
1359422495723, }, attemptId: 1, }, id: 2, }, state: C_RUNNING, diagnostics:
"Container killed by the ApplicationMaster.\n", exit_status: -1000,
        DefaultContainerExecutor: Exit code from task is : 143
NMAuditLogger: USER=hadoop OPERATION=Container Finished - Killed
TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1359422495723_0001
CONTAINERID=container_1359422495723_0001_01_000002

My questions:
        1. Why  were 2 containers created in Hadoop02,however Hadoop04 got
nothing.is it normal ?
2. What is the principle that guides containers to be created.
3. Why were the two containers (the container_*_000003 and the
container_*_000002)  killed, while the container_*_000001 succeeded.
   is it normal?
logs of Hadoop01 as follows:

2013-01-29 09:23:48,904 INFO
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated
new applicationId: 1
2013-01-29 09:23:50,201 INFO
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application
with id 1 submitted by user hadoop
2013-01-29 09:23:50,204 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop
IP=10.167.14.221 OPERATION=Submit Application Request
TARGET=ClientRMServiceRESULT=SUCCESS APPID=application_1359422495723_0001
2013-01-29 09:23:50,221 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1359422495723_0001 State change from NEW to SUBMITTED
2013-01-29 09:23:50,221 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering appattempt_1359422495723_0001_000001
2013-01-29 09:23:50,222 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1359422495723_0001_000001 State change from NEW to SUBMITTED
2013-01-29 09:23:50,242 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler:
Application Submission: application_1359422495723_0001 from hadoop,
currently active: 1
2013-01-29 09:23:50,250 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1359422495723_0001_000001 State change from SUBMITTED to
SCHEDULED
2013-01-29 09:23:50,250 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1359422495723_0001 State change from SUBMITTED to ACCEPTED
2013-01-29 09:23:50,581 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1359422495723_0001_01_000001 Container Transitioned from NEW to
ALLOCATED
2013-01-29 09:23:50,581 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
USER=hadoop OPERATION=AM
Allocated Container TARGET=SchedulerApp RESULT=SUCCESS
APPID=application_1359422495723_0001
CONTAINERID=container_1359422495723_0001_01_000001
2013-01-29 09:23:50,581 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1359422495723_0001_01_000001 of capacity
memory: 1536 on host Hadoop02:39876, which currently has 1 containers,
memory: 1536 used and memory: 6656 available
2013-01-29 09:23:50,582 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1359422495723_0001_01_000001 Container Transitioned from
ALLOCATED to ACQUIRED
2013-01-29 09:23:50,583 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1359422495723_0001_000001 State change from SCHEDULED to
ALLOCATED
2013-01-29 09:23:50,587 INFO
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher:
Launching masterappattempt_1359422495723_0001_000001
2013-01-29 09:23:50,606 INFO
org.apache.hadoop.yarn.s
+
YouPeng Yang 2013-02-02, 14:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB