Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Re: setLocalResources() on ContainerLaunchContext


Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Krishna Kishore Bonagiri 2013-08-06, 12:25
I tried the following and it works!
String shellScriptPath = "file:///home_/dsadm/kishore/kk.ksh";

But now getting a timestamp error like below, when I passed 0 to
setTimestamp()

13/08/06 08:23:48 INFO ApplicationMaster: Got container status for
containerID= container_1375784329048_0017_01_000002, state=COMPLETE,
exitStatus=-1000, diagnostics=Resource file:/home_/dsadm/kishore/kk.ksh
changed on src filesystem (expected 0, was 1367580580000
On Tue, Aug 6, 2013 at 5:24 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Can you try passing a fully qualified local path? That is, including the
> file:/ scheme
> On Aug 6, 2013 4:05 PM, "Krishna Kishore Bonagiri" <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Harsh,
>>    The setResource() call on LocalResource() is expecting an argument of
>> type org.apache.hadoop.yarn.api.records.URL which is converted from a
>> string in the form of URI. This happens in the following call of
>> Distributed Shell example,
>>
>> shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new URI(
>> shellScriptPath)));
>>
>> So, if I give a local file I get a parsing error like below, which is
>> when I changed it to an HDFS file thinking that it should be given like
>> that only. Could you please give an example of how else it could be used,
>> using a local file as you are saying?
>>
>> 2013-08-06 06:23:12,942 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Failed to parse resource-request
>> java.net.URISyntaxException: Expected scheme name at index 0:
>> :///home_/dsadm/kishore/kk.ksh
>>         at java.net.URI$Parser.fail(URI.java:2820)
>>         at java.net.URI$Parser.failExpecting(URI.java:2826)
>>         at java.net.URI$Parser.parse(URI.java:3015)
>>         at java.net.URI.<init>(URI.java:747)
>>         at
>> org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:77)
>>         at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.<init>(LocalResourceRequest.java:46)
>>
>>
>>
>> On Tue, Aug 6, 2013 at 3:36 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> To be honest, I've never tried loading a HDFS file onto the
>>> LocalResource this way. I usually just pass a local file and that
>>> works just fine. There may be something in the URI transformation
>>> possibly breaking a HDFS source, but try passing a local file - does
>>> that fail too? The Shell example uses a local file.
>>>
>>> On Tue, Aug 6, 2013 at 10:54 AM, Krishna Kishore Bonagiri
>>> <[EMAIL PROTECTED]> wrote:
>>> > Hi Harsh,
>>> >
>>> >   Please see if this is useful, I got a stack trace after the error has
>>> > occurred....
>>> >
>>> > 2013-08-06 00:55:30,559 INFO
>>> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
>>> CWD set
>>> > to
>>> /tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
>>> > >>> >
>>> file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
>>> > 2013-08-06 00:55:31,017 ERROR
>>> > org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException
>>> > as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does
>>> not
>>> > exist: hdfs://isredeng/kishore/kk.ksh
>>> > 2013-08-06 00:55:31,029 INFO
>>> >
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>>> > DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File
>>> does
>>> > not exist: hdfs://isredeng/kishore/kk.ksh
>>> > 2013-08-06 00:55:31,031 INFO
>>> >
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>>> > Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING
>>> to
>>> > FAILED
>>> > 2013-08-06 00:55:31,034 INFO
>>> >
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>>> > Container container_1375716148174_0004_01_000002 transitioned from
>>> > LOCALIZING to LOCALIZATION_FAILED
>>> > 2013-08-06 00:55:31,035 INFO
+
Omkar Joshi 2013-08-06, 18:11
+
Hitesh Shah 2013-08-06, 19:05