Hi

I am trying to get the ApplicationMaster info using the <http://<proxy http
address:port>/proxy/{appid}/ws/v1/mapreduce/info> link as described on the <
http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html>
page.

I am able to access and retrieve JSON response for other modules
(ResourceManager, NodeManager and HistoryServer). However, I am getting
'Page not found' when I try to use my ResourceManager Http address to
access the ApplicationMaster info. I am using <
http://localhost:8088/proxy/{appid}/ws/v1/mapreduce/info> to retrieve JSON
response.

The instructions say "The application master should be accessed via the
proxy. This proxy is configurable to run either on the resource manager or
on a separate host."

My yarn-default.xml contains:
  <property>
    <description>The address for the web proxy as HOST:PORT, if this is not
     given then the proxy will run as part of the RM</description>
     <name>yarn.web-proxy.address</name>
     <value/>
  </property>

and I did not set a value explicitly in yarn-site.xml.  Any idea how I can
get this working? Thanks in advance.

Regards,
Prajakta

As far as I can tell, the MR WebApp, as the name itself indicates on
its doc page, starts only at the MR AM (which may be running at any
NM), and it starts as an ephemeral port logged at in the AM logs
usually as:

INFO Web app /mapreduce started at [PORT]

That it starts its own server with an ephemeral access point makes
sense, since each job uses its own AM and having a common location may
not work with the form of REST API documented at your link. Can you
please file a JIRA to fix the doc and remove the proxy server refs,
which are misleading?

Do correct me if I'm wrong.

On Thu, Jun 28, 2012 at 6:13 PM, Prajakta Kalmegh <[EMAIL PROTECTED]> wrote:
> Hi
>
> I am trying to get the ApplicationMaster info using the <http://<proxy http
> address:port>/proxy/{appid}/ws/v1/mapreduce/info> link as described on the <
> http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html>
> page.
>
> I am able to access and retrieve JSON response for other modules
> (ResourceManager, NodeManager and HistoryServer). However, I am getting
> 'Page not found' when I try to use my ResourceManager Http address to
> access the ApplicationMaster info. I am using <
> http://localhost:8088/proxy/{appid}/ws/v1/mapreduce/info> to retrieve JSON
> response.
>
> The instructions say "The application master should be accessed via the
> proxy. This proxy is configurable to run either on the resource manager or
> on a separate host."
>
> My yarn-default.xml contains:
>  <property>
>    <description>The address for the web proxy as HOST:PORT, if this is not
>     given then the proxy will run as part of the RM</description>
>     <name>yarn.web-proxy.address</name>
>     <value/>
>  </property>
>
> and I did not set a value explicitly in yarn-site.xml.  Any idea how I can
> get this working? Thanks in advance.
>
> Regards,
> Prajakta

--
Harsh J

Hi Harsh

Can you tell me where can I find my AM logs? I could find the
RM/NM/HistoryServer/Container logs but not AM logs :(

I looked in the RM logs to see if I have any entry for 'Web app /mapreduce'
to see if it actually starts on RM as indicated in the link, but it does
not.

I will file a JIRA once I know where the default value is set for the same.
Maybe instead of removing the misleading links, we can fix them once I know
how to retrieve data from the ApplicationMaster URL.

Regards,
Prajakta
On Thu, Jun 28, 2012 at 8:16 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> As far as I can tell, the MR WebApp, as the name itself indicates on
> its doc page, starts only at the MR AM (which may be running at any
> NM), and it starts as an ephemeral port logged at in the AM logs
> usually as:
>
> INFO Web app /mapreduce started at [PORT]
>
> That it starts its own server with an ephemeral access point makes
> sense, since each job uses its own AM and having a common location may
> not work with the form of REST API documented at your link. Can you
> please file a JIRA to fix the doc and remove the proxy server refs,
> which are misleading?
>
> Do correct me if I'm wrong.
>
> On Thu, Jun 28, 2012 at 6:13 PM, Prajakta Kalmegh <[EMAIL PROTECTED]>
> wrote:
> > Hi
> >
> > I am trying to get the ApplicationMaster info using the <http://<proxy
> http
> > address:port>/proxy/{appid}/ws/v1/mapreduce/info> link as described on
> the <
> >
> http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html
> >
> > page.
> >
> > I am able to access and retrieve JSON response for other modules
> > (ResourceManager, NodeManager and HistoryServer). However, I am getting
> > 'Page not found' when I try to use my ResourceManager Http address to
> > access the ApplicationMaster info. I am using <
> > http://localhost:8088/proxy/{appid}/ws/v1/mapreduce/info> to retrieve
> JSON
> > response.
> >
> > The instructions say "The application master should be accessed via the
> > proxy. This proxy is configurable to run either on the resource manager
> or
> > on a separate host."
> >
> > My yarn-default.xml contains:
> >  <property>
> >    <description>The address for the web proxy as HOST:PORT, if this is
> not
> >     given then the proxy will run as part of the RM</description>
> >     <name>yarn.web-proxy.address</name>
> >     <value/>
> >  </property>
> >
> > and I did not set a value explicitly in yarn-site.xml.  Any idea how I
> can
> > get this working? Thanks in advance.
> >
> > Regards,
> > Prajakta
>
>
>
> --
> Harsh J
>

Please don't file that JIRA.  The proxy server is intended to front the
web server for all calls to the AM.  This is so you only have to go to a
single location to get to any AM's web service.  The proxy server is a
very simple proxy and just forwards the extra part of the path on to the
AM.

If you are having issues with this please include the version you are
having problems with.  Also please look at the logs for the RM on startup
to see if there is anything there indicating why it is not starting up.

--Bobby Evans

On 6/28/12 9:46 AM, "Harsh J" <[EMAIL PROTECTED]> wrote:

>As far as I can tell, the MR WebApp, as the name itself indicates on
>its doc page, starts only at the MR AM (which may be running at any
>NM), and it starts as an ephemeral port logged at in the AM logs
>usually as:
>
>INFO Web app /mapreduce started at [PORT]
>
>That it starts its own server with an ephemeral access point makes
>sense, since each job uses its own AM and having a common location may
>not work with the form of REST API documented at your link. Can you
>please file a JIRA to fix the doc and remove the proxy server refs,
>which are misleading?
>
>Do correct me if I'm wrong.
>
>On Thu, Jun 28, 2012 at 6:13 PM, Prajakta Kalmegh <[EMAIL PROTECTED]>
>wrote:
>> Hi
>>
>> I am trying to get the ApplicationMaster info using the <http://<proxy
>>http
>> address:port>/proxy/{appid}/ws/v1/mapreduce/info> link as described on
>>the <
>>
>>http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn
>>-site/MapredAppMasterRest.html>
>> page.
>>
>> I am able to access and retrieve JSON response for other modules
>> (ResourceManager, NodeManager and HistoryServer). However, I am getting
>> 'Page not found' when I try to use my ResourceManager Http address to
>> access the ApplicationMaster info. I am using <
>> http://localhost:8088/proxy/{appid}/ws/v1/mapreduce/info> to retrieve
>>JSON
>> response.
>>
>> The instructions say "The application master should be accessed via the
>> proxy. This proxy is configurable to run either on the resource manager
>>or
>> on a separate host."
>>
>> My yarn-default.xml contains:
>>  <property>
>>    <description>The address for the web proxy as HOST:PORT, if this is
>>not
>>     given then the proxy will run as part of the RM</description>
>>     <name>yarn.web-proxy.address</name>
>>     <value/>
>>  </property>
>>
>> and I did not set a value explicitly in yarn-site.xml.  Any idea how I
>>can
>> get this working? Thanks in advance.
>>
>> Regards,
>> Prajakta
>
>
>
>--
>Harsh J

Hi Robert/Harsh

Thanks for your reply.

My RM is starting just fine. The problem is with the use of
http://<proxy httpddress:port>/proxy/{appid}/ws/v1/mapreduce
to get the JSON response.

As I said before, I had not configured the yarn.web-proxy.address
property in yarn-site.xml. I assumed it will use the RM's
yarn.resourcemanager.webapp.address property value as default.
However, it gives me a '404-Page not found error'.  Today I tried
specifying a value explicitly for the yarn.web-proxy.address property.

On running the wordcount example, it even gives a url
<http://localhost:8090>/proxy/{appid}/> to track the App Mast info.
However, I am still not able to get a json response.

Also, I tried to get the data from historyserver instead of runtime
using the instructions given on page
<http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html>

HistoryServer REST response does not give me jobids corresponding to
an application. It just lists all the jobs run until now. By the way,
the documentation does say

----------

"Both of the following URI's give you the history server information,
from an application id identified by the appid value.
  * http://<history server http address:port>/ws/v1/history
  * http://<history server http address:port>/ws/v1/history/info"
---------

But there is no provision to specify the application id with these REST URLs.

Any idea how I can get the Application Master REST working and also
linking jobids to application id using the HistoryServerREST API?

Any help is appreciated. Thanks in advance.
Regards,
Prajakta
On Fri, Jun 29, 2012 at 8:55 PM, Robert Evans <[EMAIL PROTECTED]> wrote:

> Please don't file that JIRA.  The proxy server is intended to front the
> web server for all calls to the AM.  This is so you only have to go to a
> single location to get to any AM's web service.  The proxy server is a
> very simple proxy and just forwards the extra part of the path on to the
> AM.
>
> If you are having issues with this please include the version you are
> having problems with.  Also please look at the logs for the RM on startup
> to see if there is anything there indicating why it is not starting up.
>
> --Bobby Evans
>
> On 6/28/12 9:46 AM, "Harsh J" <[EMAIL PROTECTED]> wrote:
>
> >As far as I can tell, the MR WebApp, as the name itself indicates on
> >its doc page, starts only at the MR AM (which may be running at any
> >NM), and it starts as an ephemeral port logged at in the AM logs
> >usually as:
> >
> >INFO Web app /mapreduce started at [PORT]
> >
> >That it starts its own server with an ephemeral access point makes
> >sense, since each job uses its own AM and having a common location may
> >not work with the form of REST API documented at your link. Can you
> >please file a JIRA to fix the doc and remove the proxy server refs,
> >which are misleading?
> >
> >Do correct me if I'm wrong.
> >
> >On Thu, Jun 28, 2012 at 6:13 PM, Prajakta Kalmegh <[EMAIL PROTECTED]>
> >wrote:
> >> Hi
> >>
> >> I am trying to get the ApplicationMaster info using the <http://<proxy
> >>http
> >> address:port>/proxy/{appid}/ws/v1/mapreduce/info> link as described on
> >>the <
> >>
> >>
> http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn
> >>-site/MapredAppMasterRest.html>
> >> page.
> >>
> >> I am able to access and retrieve JSON response for other modules
> >> (ResourceManager, NodeManager and HistoryServer). However, I am getting
> >> 'Page not found' when I try to use my ResourceManager Http address to
> >> access the ApplicationMaster info. I am using <
> >> http://localhost:8088/proxy/{appid}/ws/v1/mapreduce/info> to retrieve
> >>JSON
> >> response.
> >>
> >> The instructions say "The application master should be accessed via the
> >> proxy. This proxy is configurable to run either on the resource manager
> >>or
> >> on a separate host."
> >>
> >> My yarn-default.xml contains:
> >>  <property>
> >>    <description>The address for the web proxy as HOST:PORT, if this is

Hi Robert

I figured out the problem just now. To avoid the below error, I had to set
the 'hadoop.http.staticuser.user' property in core-site.xml (defaults to
dr.who). I can now get runtime data from AppMaster using *curl* as well as
in GUI.

I wonder if we have to set this property even when we are not specifying
the yarn web-proxy address (when it runs as part of RM by default) as well.
If yes, was it documented somewhere which I failed to see? :(

Anyways, thanks for your response so far.

Regards,
Prajakta

On Mon, Jul 9, 2012 at 3:29 PM, Prajakta Kalmegh <[EMAIL PROTECTED]> wrote:

> Hi Robert
>
> I started the proxyserver explicitly by specifying a value for the
> yarn.web-proxy.address in yarn-site.xml. The proxyserver did start and I
> tried getting the JSON response using the following command :
>
> curl --compressed -H "Accept: application/json" -X GET "
> http://localhost:8090/proxy/application_1341823967331_0001/ws/v1/mapreduce/jobs/job_1341823967331_0001
> "
>
> However, it refused connection and below is the excerpt from the
> Proxyserver logs:
> ---------
> 2012-07-09 14:26:40,402 INFO org.mortbay.log: Extract
> jar:file:/home/prajakta/Projects/IRL/hadoop-common/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/proxy
> to /tmp/Jetty_localhost_8090_proxy____.ak3o30/webapp
> 2012-07-09 14:26:40,992 INFO org.mortbay.log: Started
> SelectChannelConnector@localhost:8090
> 2012-07-09 14:26:40,993 INFO
> org.apache.hadoop.yarn.service.AbstractService:
> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxy is started.
> 2012-07-09 14:26:40,993 INFO
> org.apache.hadoop.yarn.service.AbstractService:
> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer is started.
> 2012-07-09 14:33:26,039 INFO
> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is
> accessing unchecked
> http://prajakta:44314/ws/v1/mapreduce/jobs/job_1341823967331_0001 which
> is the app master GUI of application_1341823967331_0001 owned by prajakta
> 2012-07-09 14:33:29,277 INFO
> org.apache.commons.httpclient.HttpMethodDirector: I/O exception
> (org.apache.commons.httpclient.NoHttpResponseException) caught when
> processing request: The server prajakta failed to respond
> 2012-07-09 14:33:29,277 INFO
> org.apache.commons.httpclient.HttpMethodDirector: Retrying request
> 2012-07-09 14:33:29,284 WARN org.mortbay.log:
> /proxy/application_1341823967331_0001/ws/v1/mapreduce/jobs/job_1341823967331_0001:
> java.net.SocketException: Connection reset
> 2012-07-09 14:37:33,834 INFO
> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is
> accessing unchecked
> http://prajakta:19888/jobhistory/job/job_1341823967331_0001/jobhistory/job/job_1341823967331_0001which is the app master GUI of application_1341823967331_0001 owned by
> prajakta
> ---------------
>
> I am not sure why http request object is setting my remoteUser to dr.who.
> :(
>
> I gather from <https://issues.apache.org/jira/browse/MAPREDUCE-2858> that
> this warning is posted only in case where security is disabled. I assume
> that the proxy server is not disabled if security is disabled.
>
> Any idea what could be the reason for this I/O exception? Am I missing
> setting any property for proper access. Please let me know.
>
> Regards,
> Prajakta
>
>
>
>
>
>
> On Fri, Jul 6, 2012 at 10:59 PM, Prajakta Kalmegh <[EMAIL PROTECTED]>wrote:
>
>> I am using hadoop trunk (forked from github). It supports RESTful APIs as
>> I am able to retrieve JSON objects for RM (cluster/nodes info)+
>> Historyserver. The only issue is with AppMaster REST API.
>>
>> Regards,
>> Prajakta
>>
>>
>>
>> On Fri, Jul 6, 2012 at 10:55 PM, Robert Evans <[EMAIL PROTECTED]>wrote:
>>
>>> What version of hadoop are you using?  It could be that the version you
>>> have does not have the RESTful APIs in it yet, and the proxy is working
>>> just fine.
>>>
>>> --Bobby Evans
>>>
>>> On 7/6/12 12:06 PM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:

Hmm, that is very odd.  It only checks the user if security is enabled to
warn the user about potentially accessing something unsafe.  I am not sure
why that would cause an issue.

--Bobby Evans

On 7/9/12 6:07 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:

>Hi Robert
>
>I figured out the problem just now. To avoid the below error, I had to set
>the 'hadoop.http.staticuser.user' property in core-site.xml (defaults to
>dr.who). I can now get runtime data from AppMaster using *curl* as well as
>in GUI.
>
>I wonder if we have to set this property even when we are not specifying
>the yarn web-proxy address (when it runs as part of RM by default) as
>well.
>If yes, was it documented somewhere which I failed to see? :(
>
>Anyways, thanks for your response so far.
>
>Regards,
>Prajakta
>
>
>
>On Mon, Jul 9, 2012 at 3:29 PM, Prajakta Kalmegh <[EMAIL PROTECTED]>
>wrote:
>
>> Hi Robert
>>
>> I started the proxyserver explicitly by specifying a value for the
>> yarn.web-proxy.address in yarn-site.xml. The proxyserver did start and I
>> tried getting the JSON response using the following command :
>>
>> curl --compressed -H "Accept: application/json" -X GET "
>>
>>http://localhost:8090/proxy/application_1341823967331_0001/ws/v1/mapreduc
>>e/jobs/job_1341823967331_0001
>> "
>>
>> However, it refused connection and below is the excerpt from the
>> Proxyserver logs:
>> ---------
>> 2012-07-09 14:26:40,402 INFO org.mortbay.log: Extract
>>
>>jar:file:/home/prajakta/Projects/IRL/hadoop-common/hadoop-dist/target/had
>>oop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-yarn-common-3.0.0-SNAPSH
>>OT.jar!/webapps/proxy
>> to /tmp/Jetty_localhost_8090_proxy____.ak3o30/webapp
>> 2012-07-09 14:26:40,992 INFO org.mortbay.log: Started
>> SelectChannelConnector@localhost:8090
>> 2012-07-09 14:26:40,993 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxy is started.
>> 2012-07-09 14:26:40,993 INFO
>> org.apache.hadoop.yarn.service.AbstractService:
>> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer is
>>started.
>> 2012-07-09 14:33:26,039 INFO
>> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is
>> accessing unchecked
>> http://prajakta:44314/ws/v1/mapreduce/jobs/job_1341823967331_0001 which
>> is the app master GUI of application_1341823967331_0001 owned by
>>prajakta
>> 2012-07-09 14:33:29,277 INFO
>> org.apache.commons.httpclient.HttpMethodDirector: I/O exception
>> (org.apache.commons.httpclient.NoHttpResponseException) caught when
>> processing request: The server prajakta failed to respond
>> 2012-07-09 14:33:29,277 INFO
>> org.apache.commons.httpclient.HttpMethodDirector: Retrying request
>> 2012-07-09 14:33:29,284 WARN org.mortbay.log:
>>
>>/proxy/application_1341823967331_0001/ws/v1/mapreduce/jobs/job_1341823967
>>331_0001:
>> java.net.SocketException: Connection reset
>> 2012-07-09 14:37:33,834 INFO
>> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is
>> accessing unchecked
>>
>>http://prajakta:19888/jobhistory/job/job_1341823967331_0001/jobhistory/jo
>>b/job_1341823967331_0001which is the app master GUI of
>>application_1341823967331_0001 owned by
>> prajakta
>> ---------------
>>
>> I am not sure why http request object is setting my remoteUser to
>>dr.who.
>> :(
>>
>> I gather from <https://issues.apache.org/jira/browse/MAPREDUCE-2858>
>>that
>> this warning is posted only in case where security is disabled. I assume
>> that the proxy server is not disabled if security is disabled.
>>
>> Any idea what could be the reason for this I/O exception? Am I missing
>> setting any property for proper access. Please let me know.
>>
>> Regards,
>> Prajakta
>>
>>
>>
>>
>>
>>
>> On Fri, Jul 6, 2012 at 10:59 PM, Prajakta Kalmegh
>><[EMAIL PROTECTED]>wrote:
>>
>>> I am using hadoop trunk (forked from github). It supports RESTful APIs
>>>as
>>> I am able to retrieve JSON objects for RM (cluster/nodes info)+
>>> Historyserver. The only issue is with AppMaster REST API.

Hi Bobby

Thanks for the reply. My REST calls are working fine since I set the
'hadoop.http.staticuser.user' property to 'prajakta' instead of Dr.Who in
core-site.xml . I didn't get time to figure out the reason behind it as I
just moved on to further coding :)

Thanks,
Prajakta

On Thu, Jul 26, 2012 at 1:40 AM, Robert Evans <[EMAIL PROTECTED]> wrote:

> Hmm, that is very odd.  It only checks the user if security is enabled to
> warn the user about potentially accessing something unsafe.  I am not sure
> why that would cause an issue.
>
> --Bobby Evans
>
> On 7/9/12 6:07 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:
>
> >Hi Robert
> >
> >I figured out the problem just now. To avoid the below error, I had to set
> >the 'hadoop.http.staticuser.user' property in core-site.xml (defaults to
> >dr.who). I can now get runtime data from AppMaster using *curl* as well as
> >in GUI.
> >
> >I wonder if we have to set this property even when we are not specifying
> >the yarn web-proxy address (when it runs as part of RM by default) as
> >well.
> >If yes, was it documented somewhere which I failed to see? :(
> >
> >Anyways, thanks for your response so far.
> >
> >Regards,
> >Prajakta
> >
> >
> >
> >On Mon, Jul 9, 2012 at 3:29 PM, Prajakta Kalmegh <[EMAIL PROTECTED]>
> >wrote:
> >
> >> Hi Robert
> >>
> >> I started the proxyserver explicitly by specifying a value for the
> >> yarn.web-proxy.address in yarn-site.xml. The proxyserver did start and I
> >> tried getting the JSON response using the following command :
> >>
> >> curl --compressed -H "Accept: application/json" -X GET "
> >>
> >>
> http://localhost:8090/proxy/application_1341823967331_0001/ws/v1/mapreduc
> >>e/jobs/job_1341823967331_0001
> >> "
> >>
> >> However, it refused connection and below is the excerpt from the
> >> Proxyserver logs:
> >> ---------
> >> 2012-07-09 14:26:40,402 INFO org.mortbay.log: Extract
> >>
> >>jar:file:/home/prajakta/Projects/IRL/hadoop-common/hadoop-dist/target/had
> >>oop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-yarn-common-3.0.0-SNAPSH
> >>OT.jar!/webapps/proxy
> >> to /tmp/Jetty_localhost_8090_proxy____.ak3o30/webapp
> >> 2012-07-09 14:26:40,992 INFO org.mortbay.log: Started
> >> SelectChannelConnector@localhost:8090
> >> 2012-07-09 14:26:40,993 INFO
> >> org.apache.hadoop.yarn.service.AbstractService:
> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxy is started.
> >> 2012-07-09 14:26:40,993 INFO
> >> org.apache.hadoop.yarn.service.AbstractService:
> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer is
> >>started.
> >> 2012-07-09 14:33:26,039 INFO
> >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is
> >> accessing unchecked
> >> http://prajakta:44314/ws/v1/mapreduce/jobs/job_1341823967331_0001 which
> >> is the app master GUI of application_1341823967331_0001 owned by
> >>prajakta
> >> 2012-07-09 14:33:29,277 INFO
> >> org.apache.commons.httpclient.HttpMethodDirector: I/O exception
> >> (org.apache.commons.httpclient.NoHttpResponseException) caught when
> >> processing request: The server prajakta failed to respond
> >> 2012-07-09 14:33:29,277 INFO
> >> org.apache.commons.httpclient.HttpMethodDirector: Retrying request
> >> 2012-07-09 14:33:29,284 WARN org.mortbay.log:
> >>
> >>/proxy/application_1341823967331_0001/ws/v1/mapreduce/jobs/job_1341823967
> >>331_0001:
> >> java.net.SocketException: Connection reset
> >> 2012-07-09 14:37:33,834 INFO
> >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is
> >> accessing unchecked
> >>
> >>
> http://prajakta:19888/jobhistory/job/job_1341823967331_0001/jobhistory/jo
> >>b/job_1341823967331_0001which is the app master GUI of
> >>application_1341823967331_0001 owned by
> >> prajakta
> >> ---------------
> >>
> >> I am not sure why http request object is setting my remoteUser to
> >>dr.who.
> >> :(
> >>
> >> I gather from <https://issues.apache.org/jira/browse/MAPREDUCE-2858>
> >>that
> >> this warning is posted only in case where security is disabled. I assume

OK I think I understand it now.  You probably have ACLs enabled, but no
web filter on the RM to let you sign in as a given user.  As such the
default filter is making you be Dr. Who, or whomever else it is, but the
ACL check in the web service is rejecting Dr Who, because that is not the
correct user.  You will probably run into this issue again if anyone else
but you runs something.  You could fix this by either disabling the ACL
check, which makes a lot of since for a cluster without security, or you
could implement a servlet Filter for the RM that would let you sign on as
a given user.

--Bobby Evans
On 7/26/12 12:48 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:

>Hi Bobby
>
>Thanks for the reply. My REST calls are working fine since I set the
>'hadoop.http.staticuser.user' property to 'prajakta' instead of Dr.Who in
>core-site.xml . I didn't get time to figure out the reason behind it as I
>just moved on to further coding :)
>
>Thanks,
>Prajakta
>
>
>
>On Thu, Jul 26, 2012 at 1:40 AM, Robert Evans <[EMAIL PROTECTED]> wrote:
>
>> Hmm, that is very odd.  It only checks the user if security is enabled
>>to
>> warn the user about potentially accessing something unsafe.  I am not
>>sure
>> why that would cause an issue.
>>
>> --Bobby Evans
>>
>> On 7/9/12 6:07 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:
>>
>> >Hi Robert
>> >
>> >I figured out the problem just now. To avoid the below error, I had to
>>set
>> >the 'hadoop.http.staticuser.user' property in core-site.xml (defaults
>>to
>> >dr.who). I can now get runtime data from AppMaster using *curl* as
>>well as
>> >in GUI.
>> >
>> >I wonder if we have to set this property even when we are not
>>specifying
>> >the yarn web-proxy address (when it runs as part of RM by default) as
>> >well.
>> >If yes, was it documented somewhere which I failed to see? :(
>> >
>> >Anyways, thanks for your response so far.
>> >
>> >Regards,
>> >Prajakta
>> >
>> >
>> >
>> >On Mon, Jul 9, 2012 at 3:29 PM, Prajakta Kalmegh <[EMAIL PROTECTED]>
>> >wrote:
>> >
>> >> Hi Robert
>> >>
>> >> I started the proxyserver explicitly by specifying a value for the
>> >> yarn.web-proxy.address in yarn-site.xml. The proxyserver did start
>>and I
>> >> tried getting the JSON response using the following command :
>> >>
>> >> curl --compressed -H "Accept: application/json" -X GET "
>> >>
>> >>
>>
>>http://localhost:8090/proxy/application_1341823967331_0001/ws/v1/mapreduc
>> >>e/jobs/job_1341823967331_0001
>> >> "
>> >>
>> >> However, it refused connection and below is the excerpt from the
>> >> Proxyserver logs:
>> >> ---------
>> >> 2012-07-09 14:26:40,402 INFO org.mortbay.log: Extract
>> >>
>>
>>>>jar:file:/home/prajakta/Projects/IRL/hadoop-common/hadoop-dist/target/h
>>>>ad
>>
>>>>oop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-yarn-common-3.0.0-SNAP
>>>>SH
>> >>OT.jar!/webapps/proxy
>> >> to /tmp/Jetty_localhost_8090_proxy____.ak3o30/webapp
>> >> 2012-07-09 14:26:40,992 INFO org.mortbay.log: Started
>> >> SelectChannelConnector@localhost:8090
>> >> 2012-07-09 14:26:40,993 INFO
>> >> org.apache.hadoop.yarn.service.AbstractService:
>> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxy is
>>started.
>> >> 2012-07-09 14:26:40,993 INFO
>> >> org.apache.hadoop.yarn.service.AbstractService:
>> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer is
>> >>started.
>> >> 2012-07-09 14:33:26,039 INFO
>> >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is
>> >> accessing unchecked
>> >> http://prajakta:44314/ws/v1/mapreduce/jobs/job_1341823967331_0001
>>which
>> >> is the app master GUI of application_1341823967331_0001 owned by
>> >>prajakta
>> >> 2012-07-09 14:33:29,277 INFO
>> >> org.apache.commons.httpclient.HttpMethodDirector: I/O exception
>> >> (org.apache.commons.httpclient.NoHttpResponseException) caught when
>> >> processing request: The server prajakta failed to respond
>> >> 2012-07-09 14:33:29,277 INFO
>> >> org.apache.commons.httpclient.HttpMethodDirector: Retrying request

:) Yes, you are right. The yarn.acl.enable property in yarn-default.xml is
set true. If the property is true by default, then this makes it mandatory
for users to either specify a value for hadoop.http.staticuser.user
property explicitly or to change the acl's to false. Am I right to assume
this?

Regards,
Prajakta

On Thu, Jul 26, 2012 at 11:59 PM, Robert Evans <[EMAIL PROTECTED]> wrote:

> OK I think I understand it now.  You probably have ACLs enabled, but no
> web filter on the RM to let you sign in as a given user.  As such the
> default filter is making you be Dr. Who, or whomever else it is, but the
> ACL check in the web service is rejecting Dr Who, because that is not the
> correct user.  You will probably run into this issue again if anyone else
> but you runs something.  You could fix this by either disabling the ACL
> check, which makes a lot of since for a cluster without security, or you
> could implement a servlet Filter for the RM that would let you sign on as
> a given user.
>
> --Bobby Evans
>
>
> On 7/26/12 12:48 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:
>
> >Hi Bobby
> >
> >Thanks for the reply. My REST calls are working fine since I set the
> >'hadoop.http.staticuser.user' property to 'prajakta' instead of Dr.Who in
> >core-site.xml . I didn't get time to figure out the reason behind it as I
> >just moved on to further coding :)
> >
> >Thanks,
> >Prajakta
> >
> >
> >
> >On Thu, Jul 26, 2012 at 1:40 AM, Robert Evans <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hmm, that is very odd.  It only checks the user if security is enabled
> >>to
> >> warn the user about potentially accessing something unsafe.  I am not
> >>sure
> >> why that would cause an issue.
> >>
> >> --Bobby Evans
> >>
> >> On 7/9/12 6:07 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:
> >>
> >> >Hi Robert
> >> >
> >> >I figured out the problem just now. To avoid the below error, I had to
> >>set
> >> >the 'hadoop.http.staticuser.user' property in core-site.xml (defaults
> >>to
> >> >dr.who). I can now get runtime data from AppMaster using *curl* as
> >>well as
> >> >in GUI.
> >> >
> >> >I wonder if we have to set this property even when we are not
> >>specifying
> >> >the yarn web-proxy address (when it runs as part of RM by default) as
> >> >well.
> >> >If yes, was it documented somewhere which I failed to see? :(
> >> >
> >> >Anyways, thanks for your response so far.
> >> >
> >> >Regards,
> >> >Prajakta
> >> >
> >> >
> >> >
> >> >On Mon, Jul 9, 2012 at 3:29 PM, Prajakta Kalmegh <[EMAIL PROTECTED]>
> >> >wrote:
> >> >
> >> >> Hi Robert
> >> >>
> >> >> I started the proxyserver explicitly by specifying a value for the
> >> >> yarn.web-proxy.address in yarn-site.xml. The proxyserver did start
> >>and I
> >> >> tried getting the JSON response using the following command :
> >> >>
> >> >> curl --compressed -H "Accept: application/json" -X GET "
> >> >>
> >> >>
> >>
> >>
> http://localhost:8090/proxy/application_1341823967331_0001/ws/v1/mapreduc
> >> >>e/jobs/job_1341823967331_0001
> >> >> "
> >> >>
> >> >> However, it refused connection and below is the excerpt from the
> >> >> Proxyserver logs:
> >> >> ---------
> >> >> 2012-07-09 14:26:40,402 INFO org.mortbay.log: Extract
> >> >>
> >>
> >>>>jar:file:/home/prajakta/Projects/IRL/hadoop-common/hadoop-dist/target/h
> >>>>ad
> >>
> >>>>oop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-yarn-common-3.0.0-SNAP
> >>>>SH
> >> >>OT.jar!/webapps/proxy
> >> >> to /tmp/Jetty_localhost_8090_proxy____.ak3o30/webapp
> >> >> 2012-07-09 14:26:40,992 INFO org.mortbay.log: Started
> >> >> SelectChannelConnector@localhost:8090
> >> >> 2012-07-09 14:26:40,993 INFO
> >> >> org.apache.hadoop.yarn.service.AbstractService:
> >> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxy is
> >>started.
> >> >> 2012-07-09 14:26:40,993 INFO
> >> >> org.apache.hadoop.yarn.service.AbstractService:
> >> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer is
> >> >>started.
> >> >> 2012-07-09 14:33:26,039 INFO

Yes you are right.  If it is true by default we probably want to update
the documentation for the web services to indicate this.  Could you file a
JIRA for improving that documentation?

Thanks,

Bobby

On 7/27/12 3:11 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:

>:) Yes, you are right. The yarn.acl.enable property in yarn-default.xml is
>set true. If the property is true by default, then this makes it mandatory
>for users to either specify a value for hadoop.http.staticuser.user
>property explicitly or to change the acl's to false. Am I right to assume
>this?
>
>Regards,
>Prajakta
>
>
>
>On Thu, Jul 26, 2012 at 11:59 PM, Robert Evans <[EMAIL PROTECTED]>
>wrote:
>
>> OK I think I understand it now.  You probably have ACLs enabled, but no
>> web filter on the RM to let you sign in as a given user.  As such the
>> default filter is making you be Dr. Who, or whomever else it is, but the
>> ACL check in the web service is rejecting Dr Who, because that is not
>>the
>> correct user.  You will probably run into this issue again if anyone
>>else
>> but you runs something.  You could fix this by either disabling the ACL
>> check, which makes a lot of since for a cluster without security, or you
>> could implement a servlet Filter for the RM that would let you sign on
>>as
>> a given user.
>>
>> --Bobby Evans
>>
>>
>> On 7/26/12 12:48 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:
>>
>> >Hi Bobby
>> >
>> >Thanks for the reply. My REST calls are working fine since I set the
>> >'hadoop.http.staticuser.user' property to 'prajakta' instead of Dr.Who
>>in
>> >core-site.xml . I didn't get time to figure out the reason behind it
>>as I
>> >just moved on to further coding :)
>> >
>> >Thanks,
>> >Prajakta
>> >
>> >
>> >
>> >On Thu, Jul 26, 2012 at 1:40 AM, Robert Evans <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Hmm, that is very odd.  It only checks the user if security is
>>enabled
>> >>to
>> >> warn the user about potentially accessing something unsafe.  I am not
>> >>sure
>> >> why that would cause an issue.
>> >>
>> >> --Bobby Evans
>> >>
>> >> On 7/9/12 6:07 AM, "Prajakta Kalmegh" <[EMAIL PROTECTED]> wrote:
>> >>
>> >> >Hi Robert
>> >> >
>> >> >I figured out the problem just now. To avoid the below error, I had
>>to
>> >>set
>> >> >the 'hadoop.http.staticuser.user' property in core-site.xml
>>(defaults
>> >>to
>> >> >dr.who). I can now get runtime data from AppMaster using *curl* as
>> >>well as
>> >> >in GUI.
>> >> >
>> >> >I wonder if we have to set this property even when we are not
>> >>specifying
>> >> >the yarn web-proxy address (when it runs as part of RM by default)
>>as
>> >> >well.
>> >> >If yes, was it documented somewhere which I failed to see? :(
>> >> >
>> >> >Anyways, thanks for your response so far.
>> >> >
>> >> >Regards,
>> >> >Prajakta
>> >> >
>> >> >
>> >> >
>> >> >On Mon, Jul 9, 2012 at 3:29 PM, Prajakta Kalmegh
>><[EMAIL PROTECTED]>
>> >> >wrote:
>> >> >
>> >> >> Hi Robert
>> >> >>
>> >> >> I started the proxyserver explicitly by specifying a value for the
>> >> >> yarn.web-proxy.address in yarn-site.xml. The proxyserver did start
>> >>and I
>> >> >> tried getting the JSON response using the following command :
>> >> >>
>> >> >> curl --compressed -H "Accept: application/json" -X GET "
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>http://localhost:8090/proxy/application_1341823967331_0001/ws/v1/mapreduc
>> >> >>e/jobs/job_1341823967331_0001
>> >> >> "
>> >> >>
>> >> >> However, it refused connection and below is the excerpt from the
>> >> >> Proxyserver logs:
>> >> >> ---------
>> >> >> 2012-07-09 14:26:40,402 INFO org.mortbay.log: Extract
>> >> >>
>> >>
>>
>>>>>>jar:file:/home/prajakta/Projects/IRL/hadoop-common/hadoop-dist/target
>>>>>>/h
>> >>>>ad
>> >>
>>
>>>>>>oop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-yarn-common-3.0.0-SN
>>>>>>AP
>> >>>>SH
>> >> >>OT.jar!/webapps/proxy
>> >> >> to /tmp/Jetty_localhost_8090_proxy____.ak3o30/webapp
>> >> >> 2012-07-09 14:26:40,992 INFO org.mortbay.log: Started
>> >> >> SelectChannelConnector@localhost:8090