Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> FW: meaning of cryptic yarn nodemanager message: org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices


Copy link to this message
-
RE: meaning of cryptic yarn nodemanager message: org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
RE: meaning of cryptic yarn nodemanager message: org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices

I answered one of my own questions:
> Should I change the name be changed to "httpshuffle"  or not ?
When I do change it, I get this error message and nodeserver fails to start:

2013-11-13 12:39:06,878 FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize httpshuffle
java.lang.RuntimeException: No class defined for httpshuffle

So the correct name should be 'mapreduce_shuffle' ?  
Why then the complaint in the log that it is not 'httpshuffle' ?
What issues might be caused by the name being 'mapreduce_shuffle' ?

Thanks & Regards,
Jason

-----Original Message-----
From: VasDias, Jason
Sent: 13 November 2013 12:36
To: [EMAIL PROTECTED]
Subject: meaning of cryptic yarn nodemanager message: org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices

Good day - please could anyone explain the meaning of this message in yarn-${USER}-nodemanager-${HOST}.log:

"
2013-11-13 12:17:29,704 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.
"
Love the creative spelling ! ('Auxilurary' should be 'Auxiliary' ? 'unless the refer' should be 'unless they refer' ?)

I guess this is because of the setting in yarn-site.xml :

"
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    <description>shuffle service that needs to be set for Map Reduce to run </description>  </property> "
that I installed according to the instructions at:  
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html

Why do these instructions specify this setting it may cause "issues" ? What sort of issues may be caused ?
This sentence is nonsensical:
  " Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config."
What are the "tools" being referred to here ?
What "may have issues" ?
What are the "issues" ?
What  must refer to the name in the config ?
Should I change the name to "httpshuffle"  or not ?

Any ideas / suggestions gratefully received.

Regards,
Jason
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB