Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: setLocalResources() on ContainerLaunchContext


Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Hi Harsh, Hitesh & Omkar,

  Thanks for the replies.

I tried getting the last modified timestamp like this and it works. Is this
a right thing to do?

      File file = new File("/home_/dsadm/kishore/kk.ksh");
      shellRsrc.setTimestamp(file.lastModified());
And, when I tried using a hdfs file qualifying it with both node name and
port, it didn't work, I get a similar error as earlier.

      String shellScriptPath = "hdfs://isredeng:8020//kishore/kk.ksh";
13/08/07 01:36:28 INFO ApplicationMaster: Got container status for
containerID= container_1375853431091_0005_01_000002, state=COMPLETE,
exitStatus=-1000, diagnostics=File does not exist:
hdfs://isredeng:8020/kishore/kk.ksh

13/08/07 01:36:28 INFO ApplicationMaster: Got failure status for a
container : -1000

On Wed, Aug 7, 2013 at 7:45 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Thanks Hitesh!
>
> P.s. Port isn't a requirement (and with HA URIs, you shouldn't add a
> port), but "isredeng" has to be the authority component.
>
> On Wed, Aug 7, 2013 at 7:37 AM, Hitesh Shah <[EMAIL PROTECTED]> wrote:
> > @Krishna, your logs showed the file error for
> "hdfs://isredeng/kishore/kk.ksh"
> >
> > I am assuming you have tried dfs -ls /kishore/kk.ksh and confirmed that
> the file exists? Also the qualified path seems to be missing the namenode
> port. I need to go back and check if a path without the port works by
> assuming the default namenode port.
> >
> > @Harsh, adding a helper function seems like a good idea. Let me file a
> jira to have the above added to one of the helper/client libraries.
> >
> > thanks
> > -- Hitesh
> >
> > On Aug 6, 2013, at 6:47 PM, Harsh J wrote:
> >
> >> It is kinda unnecessary to be asking developers to load in timestamps
> and
> >> length themselves. Why not provide a java.io.File, or perhaps a Path
> >> accepting API, that gets it automatically on their behalf using the
> >> FileSystem API internally?
> >>
> >> P.s. A HDFS file gave him a FNF, while a Local file gave him a proper
> >> TS/Len error. I'm guessing there's a bug here w.r.t. handling HDFS
> >> paths.
> >>
> >> On Wed, Aug 7, 2013 at 12:35 AM, Hitesh Shah <[EMAIL PROTECTED]> wrote:
> >>> Hi Krishna,
> >>>
> >>> YARN downloads a specified local resource on the container's node from
> the url specified. In all situtations, the remote url needs to be a fully
> qualified path. To verify that the file at the remote url is still valid,
> YARN expects you to provide the length and last modified timestamp of that
> file.
> >>>
> >>> If you use an hdfs path such as hdfs://namenode:port/<absolute path to
> file>, you will need to get the length and timestamp from HDFS.
> >>> If you use file:///, the file should exist on all nodes and all nodes
> should have the file with the same length and timestamp for localization to
> work. ( For a single node setup, this works but tougher to get right on a
> multi-node setup - deploying the file via a rpm should likely work).
> >>>
> >>> -- Hitesh
> >>>
> >>> On Aug 6, 2013, at 11:11 AM, Omkar Joshi wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> You need to match the timestamp. Probably get the timestamp locally
> before adding it. This is explicitly done to ensure that file is not
> updated after user makes the call to avoid possible errors.
> >>>>
> >>>>
> >>>> Thanks,
> >>>> Omkar Joshi
> >>>> Hortonworks Inc.
> >>>>
> >>>>
> >>>> On Tue, Aug 6, 2013 at 5:25 AM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
> >>>> I tried the following and it works!
> >>>> String shellScriptPath = "file:///home_/dsadm/kishore/kk.ksh";
> >>>>
> >>>> But now getting a timestamp error like below, when I passed 0 to
> setTimestamp()
> >>>>
> >>>> 13/08/06 08:23:48 INFO ApplicationMaster: Got container status for
> containerID= container_1375784329048_0017_01_000002, state=COMPLETE,
> exitStatus=-1000, diagnostics=Resource file:/home_/dsadm/kishore/kk.ksh
> changed on src filesystem (expected 0, was 1367580580000
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Aug 6, 2013 at 5:24 PM, Harsh J <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB