Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - RE: Debugging YARN AM


+
Devaraj k 2013-06-18, 06:38
+
Curtis Ullerich 2013-06-18, 19:32
Copy link to this message
-
Re: Debugging YARN AM
Alejandro Abdelnur 2013-06-18, 21:29
If distributed shell is running as an unmanaged AM then you should set the
debug flags for the 'hadoop jar' invocation, doing an 'export
HADOOP_OPTS=....' with the debug flags would do.

Thx
On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <[EMAIL PROTECTED]>wrote:

> Update: It looks like I could add the flag at line 515 of
> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
> (package org.apache.hadoop.yarn.applications.distributedshell).
>
> I tried this:
>     vargs.add("-Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>
> I recompiled the jar and launched it with
> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
> whoami
>
> I added a logging statement to make sure the change took effect. It
> prints, but it doesn't seem to halt like the MR example (because of the
> suspend=y option) even after launching the debugger. Why might this be?
>
> Thanks!
> Curtis
>
>
>
> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <[EMAIL PROTECTED]>wrote:
>
>> Hi Devaraj,
>>
>> That's what I'm not sure how to do. I want to debug by connecting
>> remotely, and I'm not sure how to configure that in the code. I haven't
>> found anything telling in the docs/source. Can you point me in the right
>> direction?
>>
>> Thanks,
>> Curtis
>>
>>
>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <[EMAIL PROTECTED]> wrote:
>>
>>>  Hi Curtis, ****
>>>
>>> ** **
>>>
>>>      "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>> MRAppMaster. It is not applicable for DistributedShell AM.  ****
>>>
>>> ** **
>>>
>>> If you want to dump out debug information then you can make use of the
>>> debug option of DistributedShell application. If you want to debug by
>>> connecting remotely, you need to update the DS application code accordingly.
>>> ****
>>>
>>> ** **
>>>
>>> Thanks****
>>>
>>> Devaraj K****
>>>
>>> ** **
>>>
>>> *From:* Curtis Ullerich [mailto:[EMAIL PROTECTED]]
>>> *Sent:* 18 June 2013 08:19
>>> *To:* [EMAIL PROTECTED]
>>> *Subject:* Debugging YARN AM****
>>>
>>> ** **
>>>
>>> Hi all, ****
>>>
>>> ** **
>>>
>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>> mode by launching the pi estimator example with this command:****
>>>
>>> ** **
>>>
>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>>
>>> ** **
>>>
>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>> work with the DistributedShell example, presumably because it's not
>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>> learning purposes, how can I debug the DistributedShell example (and other
>>> AMs)?****
>>>
>>> ** **
>>>
>>> Thanks!****
>>>
>>> ** **
>>>
>>> Curtis****
>>>
>>
>>
>
--
Alejandro
+
Curtis Ullerich 2013-06-24, 16:17