Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Calling the JobTracker from Reducer throws InvalidCredentials GSSException


+
Harsh J 2013-09-28, 18:06
+
Manish Verma 2013-09-29, 05:21
+
Harsh J 2013-09-29, 10:51
Copy link to this message
-
Re: Calling the JobTracker from Reducer throws InvalidCredentials GSSException
Manish Verma 2013-09-29, 16:16
Hi Harsh,
Awesome, after incorporating these changes in my jobdriver, it worked for
me. It looks like I should use Oozie to launch my jobs so that I don't have
to copy this code in my driver.

Thanks for your help
Manish
On Sun, Sep 29, 2013 at 3:51 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hm, I think I forgot the bit where Oozie also adds a DT for itself to
> use:
> https://github.com/apache/oozie/blob/release-4.0.0/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L374
> .
> Doing that same additional thing to the job's driver, works just fine
> for me.
>
> On Sun, Sep 29, 2013 at 10:51 AM, Manish Verma
> <[EMAIL PROTECTED]> wrote:
> > Hi Harsh,
> >
> > Thanks for the pointer but it didn't fix my problem. Basically I am
> trying
> > to read the counters set by the mapper in my reducer. And the reducer has
> > trouble reading the mapper's counter using the JobClient. I tested this
> > logic in a single node cloudera VM which did not have kerberos installed.
> >
> > Thanks
> > Manish
> >
> >
> > On Sat, Sep 28, 2013 at 11:06 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >> You'll need to reuse the security tokens of the current job to
> >> communicate without requiring a re-auth from the task. See the
> >> specific code snippet (of what Oozie does for similar goals) at
> >>
> >>
> https://github.com/apache/oozie/blob/release-4.0.0/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java#L90
> >> and also see MAPREDUCE-4324.
> >>
> >> On Sat, Sep 28, 2013 at 2:43 AM, Manish Verma
> >> <[EMAIL PROTECTED]> wrote:
> >> > I am trying to get the job tracker counters in my reducer. It works on
> >> > single node demo hadoop but fails on a real cluster where kerberos is
> >> > used
> >> > for authentication.
> >> >
> >> > ********************************************
> >> > RunningJob parentJob > >> >                     client.getJob(JobID.forName(
> >> > context.getConfiguration().get("mapred.job.id") ));
> >> >
> >> > Counters counters = parentJob.getCounters();
> >> > ********************************************
> >> >
> >> > The call to getCounter() API throws GSSException (No valid credentials
> >> > provided - Failed to find any kerberos tgt).
> >> >
> >> > I launched this job using hadoop jar command.
> >> >
> >> > Any help would be much appreciated.
> >> >
> >> > Thanks
> >> > Manish
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>