Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Yarn HDFS and Yarn Exceptions when processing "larger" datasets.

blah blah 2013-06-26, 00:09
John Lilley 2013-07-02, 18:35
Copy link to this message
Re: Yarn HDFS and Yarn Exceptions when processing "larger" datasets.

Sorry to reply so late. I don't have the data you requested (sorry I have
no time, my deadline is within 3 days). However I have observed that this
issue occurs not only for the "larger" datasets (6.8MB), but for all
datasets and all jobs in general. However for smaller datasets (1MB) the AM
does not throw the Exception, only containers throw exceptions (same as in
previous e-mail). When these exception are throws my code (AM and
containers) does not perform any operations on HDFS, they only perform
in-memory computation and communication. Also I have observed that these
exception occur at "random", I couldn't observe any pattern. I can execute
job successfully, then resubmit the job repeating the experiment and these
exceptions occur (no change was made to src code, input dataset,or
execution/input parameters).

As for the high network usage, as I said I don't have the data. But YARN is
running on nodes which are exclusive for my experiments no other software
runs on these nodes (only OS and YARN). Besides I don't think that 20
containers working on 1MB dataset (total) can be called high network usage.


2013/6/26 Devaraj k <[EMAIL PROTECTED]>

>  Hi,****
> ** **
>    Could you check the network usage in the cluster when this problem
> occurs? Probably it is causing due to high network usage. ****
> ** **
> Thanks****
> Devaraj k****
> ** **
> *From:* blah blah [mailto:[EMAIL PROTECTED]]
> *Sent:* 26 June 2013 05:39
> *Subject:* Yarn HDFS and Yarn Exceptions when processing "larger"
> datasets.****
> ** **
> Hi All****
> First let me excuse for the poor thread title but I have no idea how to
> express the problem in one sentence. ****
> I have implemented new Application Master with the use of Yarn. I am using
> old Yarn development version. Revision 1437315, from 2013-01-23 (SNAPSHOT
> 3.0.0). I can not update to current trunk version, as prototype deadline is
> soon, and I don't have time to include Yarn API changes.****
> Currently I execute experiments in pseudo-distributed mode, I use guava
> version 14.0-rc1. I have a problem with Yarn's and HDFS Exceptions for
> "larger" datasets. My AM works fine and I can execute it without a problem
> for a debug dataset (1MB size). But when I increase the size of input to
> 6.8 MB, I am getting the following exceptions:****
> AM_Exceptions_Stack
> Exception in thread "Thread-3"
> java.lang.reflect.UndeclaredThrowableException
>     at
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
>     at
> org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.allocate(AMRMProtocolPBClientImpl.java:77)
>     at
> org.apache.hadoop.yarn.client.AMRMClientImpl.allocate(AMRMClientImpl.java:194)
>     at
> org.tudelft.ludograph.app.AppMasterContainerRequester.sendContainerAskToRM(AppMasterContainerRequester.java:219)
>     at
> org.tudelft.ludograph.app.AppMasterContainerRequester.run(AppMasterContainerRequester.java:315)
>     at java.lang.Thread.run(Thread.java:662)
> Caused by: com.google.protobuf.ServiceException: java.io.IOException:
> Failed on local exception: java.io.IOException: Response is null.; Host
> Details : local host is: "linux-ljc5.site/"; destination host
> is: "":8030;
>     at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)
>     at $Proxy10.allocate(Unknown Source)
>     at
> org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.allocate(AMRMProtocolPBClientImpl.java:75)
>     ... 4 more
> Caused by: java.io.IOException: Failed on local exception:
> java.io.IOException: Response is null.; Host Details : local host is:
> "linux-ljc5.site/"; destination host is: "":8030;
>     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:760)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1240)
>     at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)