Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: Calling the JobTracker from Reducer throws InvalidCredentials GSSException


+
Harsh J 2013-09-28, 18:06
+
Manish Verma 2013-09-29, 05:21
+
Harsh J 2013-09-29, 10:51
Copy link to this message
-
Re: Calling the JobTracker from Reducer throws InvalidCredentials GSSException
Hi Harsh,
Awesome, after incorporating these changes in my jobdriver, it worked for
me. It looks like I should use Oozie to launch my jobs so that I don't have
to copy this code in my driver.

Thanks for your help
Manish
On Sun, Sep 29, 2013 at 3:51 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hm, I think I forgot the bit where Oozie also adds a DT for itself to
> use:
> https://github.com/apache/oozie/blob/release-4.0.0/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L374
> .
> Doing that same additional thing to the job's driver, works just fine
> for me.
>
> On Sun, Sep 29, 2013 at 10:51 AM, Manish Verma
> <[EMAIL PROTECTED]> wrote:
> > Hi Harsh,
> >
> > Thanks for the pointer but it didn't fix my problem. Basically I am
> trying
> > to read the counters set by the mapper in my reducer. And the reducer has
> > trouble reading the mapper's counter using the JobClient. I tested this
> > logic in a single node cloudera VM which did not have kerberos installed.
> >
> > Thanks
> > Manish
> >
> >
> > On Sat, Sep 28, 2013 at 11:06 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >> You'll need to reuse the security tokens of the current job to
> >> communicate without requiring a re-auth from the task. See the
> >> specific code snippet (of what Oozie does for similar goals) at
> >>
> >>
> https://github.com/apache/oozie/blob/release-4.0.0/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java#L90
> >> and also see MAPREDUCE-4324.
> >>
> >> On Sat, Sep 28, 2013 at 2:43 AM, Manish Verma
> >> <[EMAIL PROTECTED]> wrote:
> >> > I am trying to get the job tracker counters in my reducer. It works on
> >> > single node demo hadoop but fails on a real cluster where kerberos is
> >> > used
> >> > for authentication.
> >> >
> >> > ********************************************
> >> > RunningJob parentJob > >> >                     client.getJob(JobID.forName(
> >> > context.getConfiguration().get("mapred.job.id") ));
> >> >
> >> > Counters counters = parentJob.getCounters();
> >> > ********************************************
> >> >
> >> > The call to getCounter() API throws GSSException (No valid credentials
> >> > provided - Failed to find any kerberos tgt).
> >> >
> >> > I launched this job using hadoop jar command.
> >> >
> >> > Any help would be much appreciated.
> >> >
> >> > Thanks
> >> > Manish
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB