Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: setLocalResources() on ContainerLaunchContext


Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Krishna Kishore Bonagiri 2013-08-07, 05:43
Hi Harsh, Hitesh & Omkar,

  Thanks for the replies.

I tried getting the last modified timestamp like this and it works. Is this
a right thing to do?

      File file = new File("/home_/dsadm/kishore/kk.ksh");
      shellRsrc.setTimestamp(file.lastModified());
And, when I tried using a hdfs file qualifying it with both node name and
port, it didn't work, I get a similar error as earlier.

      String shellScriptPath = "hdfs://isredeng:8020//kishore/kk.ksh";
13/08/07 01:36:28 INFO ApplicationMaster: Got container status for
containerID= container_1375853431091_0005_01_000002, state=COMPLETE,
exitStatus=-1000, diagnostics=File does not exist:
hdfs://isredeng:8020/kishore/kk.ksh

13/08/07 01:36:28 INFO ApplicationMaster: Got failure status for a
container : -1000

On Wed, Aug 7, 2013 at 7:45 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Thanks Hitesh!
>
> P.s. Port isn't a requirement (and with HA URIs, you shouldn't add a
> port), but "isredeng" has to be the authority component.
>
> On Wed, Aug 7, 2013 at 7:37 AM, Hitesh Shah <[EMAIL PROTECTED]> wrote:
> > @Krishna, your logs showed the file error for
> "hdfs://isredeng/kishore/kk.ksh"
> >
> > I am assuming you have tried dfs -ls /kishore/kk.ksh and confirmed that
> the file exists? Also the qualified path seems to be missing the namenode
> port. I need to go back and check if a path without the port works by
> assuming the default namenode port.
> >
> > @Harsh, adding a helper function seems like a good idea. Let me file a
> jira to have the above added to one of the helper/client libraries.
> >
> > thanks
> > -- Hitesh
> >
> > On Aug 6, 2013, at 6:47 PM, Harsh J wrote:
> >
> >> It is kinda unnecessary to be asking developers to load in timestamps
> and
> >> length themselves. Why not provide a java.io.File, or perhaps a Path
> >> accepting API, that gets it automatically on their behalf using the
> >> FileSystem API internally?
> >>
> >> P.s. A HDFS file gave him a FNF, while a Local file gave him a proper
> >> TS/Len error. I'm guessing there's a bug here w.r.t. handling HDFS
> >> paths.
> >>
> >> On Wed, Aug 7, 2013 at 12:35 AM, Hitesh Shah <[EMAIL PROTECTED]> wrote:
> >>> Hi Krishna,
> >>>
> >>> YARN downloads a specified local resource on the container's node from
> the url specified. In all situtations, the remote url needs to be a fully
> qualified path. To verify that the file at the remote url is still valid,
> YARN expects you to provide the length and last modified timestamp of that
> file.
> >>>
> >>> If you use an hdfs path such as hdfs://namenode:port/<absolute path to
> file>, you will need to get the length and timestamp from HDFS.
> >>> If you use file:///, the file should exist on all nodes and all nodes
> should have the file with the same length and timestamp for localization to
> work. ( For a single node setup, this works but tougher to get right on a
> multi-node setup - deploying the file via a rpm should likely work).
> >>>
> >>> -- Hitesh
> >>>
> >>> On Aug 6, 2013, at 11:11 AM, Omkar Joshi wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> You need to match the timestamp. Probably get the timestamp locally
> before adding it. This is explicitly done to ensure that file is not
> updated after user makes the call to avoid possible errors.
> >>>>
> >>>>
> >>>> Thanks,
> >>>> Omkar Joshi
> >>>> Hortonworks Inc.
> >>>>
> >>>>
> >>>> On Tue, Aug 6, 2013 at 5:25 AM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
> >>>> I tried the following and it works!
> >>>> String shellScriptPath = "file:///home_/dsadm/kishore/kk.ksh";
> >>>>
> >>>> But now getting a timestamp error like below, when I passed 0 to
> setTimestamp()
> >>>>
> >>>> 13/08/06 08:23:48 INFO ApplicationMaster: Got container status for
> containerID= container_1375784329048_0017_01_000002, state=COMPLETE,
> exitStatus=-1000, diagnostics=Resource file:/home_/dsadm/kishore/kk.ksh
> changed on src filesystem (expected 0, was 1367580580000
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Aug 6, 2013 at 5:24 PM, Harsh J <[EMAIL PROTECTED]> wrote: