Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: yarn-site.xml and aux-services


Copy link to this message
-
Re: yarn-site.xml and aux-services
The general practice is to install your deps into a custom location
such as /opt/john-jars, and extend YARN_CLASSPATH to include the jars,
while also configuring the classes under the aux-services list. You
need to take care of deploying jar versions to /opt/john-jars/
contents across the cluster though.

I think it may be a neat idea to have jars be placed on HDFS or any
other DFS, and the yarn-site.xml indicating the location plus class to
load. Similar to HBase co-processors. But I'll defer to Vinod on if
this would be a good thing to do.

(I know the right next thing with such an ability people will ask for
is hot-code-upgradesā€¦)

On Fri, Aug 23, 2013 at 10:11 PM, John Lilley <[EMAIL PROTECTED]> wrote:
> Are there recommended conventions for adding additional code to a stock
> Hadoop install?
>
> It would be nice if we could piggyback on whatever mechanisms are used to
> distribute hadoop itself around the cluster.
>
> john
>
>
>
> From: Vinod Kumar Vavilapalli [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 22, 2013 6:25 PM
>
>
> To: [EMAIL PROTECTED]
> Subject: Re: yarn-site.xml and aux-services
>
>
>
>
>
> Auxiliary services are essentially administer-configured services. So, they
> have to be set up at install time - before NM is started.
>
>
>
> +Vinod
>
>
>
> On Thu, Aug 22, 2013 at 1:38 PM, John Lilley <[EMAIL PROTECTED]>
> wrote:
>
> Following up on this, how exactly does one *install* the jar(s) for
> auxiliary service?  Can it be shipped out with the LocalResources of an AM?
> MapReduce's aux-service is presumably installed with Hadoop and is just
> sitting there in the right place, but if one wanted to make a whole new
> aux-service that belonged with an AM, how would one do it?
>
> John
>
>
> -----Original Message-----
> From: John Lilley [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, June 05, 2013 11:41 AM
> To: [EMAIL PROTECTED]
> Subject: RE: yarn-site.xml and aux-services
>
> Wow, thanks.  Is this documented anywhere other than the code?  I hate to
> waste y'alls time on things that can be RTFMed.
> John
>
>
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, June 05, 2013 9:35 AM
> To: <[EMAIL PROTECTED]>
> Subject: Re: yarn-site.xml and aux-services
>
> John,
>
> The format is ID and sub-config based:
>
> First, you define an ID as a service, like the string "foo". This is the ID
> the applications may lookup in their container responses map we discussed
> over another thread (around shuffle handler).
>
> <property>
>     <name>yarn.nodemanager.aux-services</name>
>     <value>foo</value>
> </property>
>
> Then you define an actual implementation class for that ID "foo", like so:
>
> <property>
> <name>yarn.nodemanager.aux-services.foo.class</name>
> <value>com.mypack.MyAuxServiceClassForFoo</value>
> </property>
>
> If you have multiple services foo and bar, then it would appear like the
> below (comma separated IDs and individual configs):
>
> <property>
>     <name>yarn.nodemanager.aux-services</name>
>     <value>foo,bar</value>
> </property>
> <property>
>     <name>yarn.nodemanager.aux-services.foo.class</name>
>     <value>com.mypack.MyAuxServiceClassForFoo</value>
> </property>
> <property>
>     <name>yarn.nodemanager.aux-services.bar.class</name>
>     <value>com.mypack.MyAuxServiceClassForBar</value>
> </property>
>
> On Wed, Jun 5, 2013 at 8:42 PM, John Lilley <[EMAIL PROTECTED]>
> wrote:
>> Good, I was hoping that would be the case.  But what are the mechanics of
>> it?  Do I just add another entry?  And what exactly is "madreduce.shuffle"?
>> A scoped class name?  Or a key string into some map elsewhere?
>>
>> e.g. like:
>>
>> <property>
>>     <name>yarn.nodemanager.aux-services</name>
>>     <value>mapreduce.shuffle</value>
>> </property>
>> <property>
>>     <name>yarn.nodemanager.aux-services</name>
>>     <value>myauxserviceclassname</value>
>> </property>
>>
>> Concerning auxiliary services -- do they communicate with NodeManager via

Harsh J
+
Harsh J 2013-09-05, 18:13
+
John Lilley 2013-09-05, 20:08