Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: Job end notification does not always work (Hadoop 2.x)


+
Ravi Prakash 2013-06-23, 13:30
+
Devaraj k 2013-06-24, 06:28
Copy link to this message
-
Re: Job end notification does not always work (Hadoop 2.x)
Ravi Prakash 2013-06-22, 21:38
Hi Prashant,

I would tend to agree with you. Although job-end notification is only a "best-effort" mechanism (i.e. we cannot always guarantee notification for example when the AM OOMs), I agree with you that we can do more. If you feel strongly about this, please create a JIRA and possibly upload a patch.

Thanks
Ravi
________________________________
 From: Prashant Kommireddi <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Thursday, June 20, 2013 9:45 PM
Subject: Job end notification does not always work (Hadoop 2.x)
 
Hello,

I came across an issue that occurs with the job notification callbacks in MR2. It works fine if the Application master has started, but does not send a callback if the initializing of AM fails.

Here is the code from MRAppMaster.java

.....
.......

// set job classloader if configured MRApps.setJobClassLoader(conf); initAndStartAppMaster(appMaster, conf, jobUserName); } catch (Throwable t) { LOG.fatal("Error starting MRAppMaster", t); System.exit(1); } }

protected static void initAndStartAppMaster(final MRAppMaster appMaster,
      final YarnConfiguration conf, String jobUserName) throws IOException,
      InterruptedException {
    UserGroupInformation.setConfiguration(conf);
    UserGroupInformation appMasterUgi = UserGroupInformation
        .createRemoteUser(jobUserName);
    appMasterUgi.doAs(new PrivilegedExceptionAction<Object>() {
      @Override
      public Object run() throws Exception {
        appMaster.init(conf);
        appMaster.start();
        if(appMaster.errorHappenedShutDown) {
          throw new IOException("Was asked to shut down.");
        }
        return null;
      }
    });
  }
appMaster.init(conf) does not dispatch JobFinishEventHandler which is responsible for sending a HTTP callback (via shutDownJob()). If there was an exception at this time, the process would simply terminate (via System.exit(1) )

appMaster.start() however rightly uses the JobFinishEventHandler and things work fine.

Shouldn't a failure on init(..) also send a callback suggesting the job failed?

Thanks,

Prashant
+
Prashant Kommireddi 2013-06-22, 21:32