Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: Job Tracker questions


Copy link to this message
-
Re: Job Tracker questions
Jeff Zhang 2010-02-09, 15:04
JobClient also use proxy of JobTracker.
On Mon, Feb 8, 2010 at 11:19 PM, Mark N <[EMAIL PROTECTED]> wrote:

> Did you check the jobClient source code?
>
>
> On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > I look at the source code, it seems the job tracker web ui also use the
> > proxy of JobTracker to get the counter information rather the xml file.
> >
> >
> > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[EMAIL PROTECTED]> wrote:
> >
> > > yes we can create a webservice in java which would be called by .net to
> > > display these counters.
> > >
> > > But since the java code to read these counters needs use hadoop APIs  (
> > job
> > > client  ) ,  am not sure we can create a webservice to read the
> counters
> > >
> > > Question is how does the default hadoop task tracker display counter
> > > information in JSP pages ? does it read from the XML files ?
> > >
> > > thanks,
> > >
> > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > >
> > > > I think you can create web service using Java, and then in .net using
> > the
> > > > web service to display the result.
> > > >
> > > >
> > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Do you mean want to connect the JobTracker using .Net ? If so, I'm
> > > afraid
> > > > I
> > > > > have no idea how to this. The rpc of hadoop is language dependent.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > >> could you please elaborate on this  ( * hint to get started  as am
> > > very
> > > > >> new
> > > > >> to hadoop? )
> > > > >> So far I could succesfully read all the default and custom
> counters.
> > > > >>
> > > > >> Currently we are having a .net client.
> > > > >>
> > > > >> thanks in advance.
> > > > >>
> > > > >>
> > > > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > wrote:
> > > > >>
> > > > >> > Well, you can create a proxy of JobTracker in client side, and
> > then
> > > > you
> > > > >> can
> > > > >> > use the API of JobTracker to get the information of jobs. The
> > Proxy
> > > > take
> > > > >> > the
> > > > >> > responsibility of  communication with the Master Node.  Read the
> > > > source
> > > > >> > code
> > > > >> > of JobClient can help you.
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]>
> > > wrote:
> > > > >> >
> > > > >> > > Ye currently am using jobclient to read these counters.
> > > > >> > >
> > > > >> > > But We are not able to use *webservices *because the jar which
> > is
> > > > used
> > > > >> to
> > > > >> > > read the counters from  running hadoop job  is itself a Hadoop
> > > > program
> > > > >> > >
> > > > >> > > If we could have pure Java Api which is run without hadoop
> > command
> > > > >> then
> > > > >> > we
> > > > >> > > could return the counter variable into webservices and show in
> > UI.
> > > > >> > >
> > > > >> > > Any help  or technique to show thsese counters in the UI would
> > be
> > > > >> > > appreciated  ( not necessarily using web service )
> > > > >> > >
> > > > >> > >
> > > > >> > > I am using webservices because I am having .net VB client
> > > > >> > >
> > > > >> > > thanks
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >> > >
> > > > >> > > > I think you can use JobClient to get the counters in your
> web
> > > > >> service.
> > > > >> > > > If you look at the shell script bin/hadoop, you will find
> that
> > > > >> actually
> > > > >> > > > this
> > > > >> > > > shell use the JobClient to get the counters.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <
> [EMAIL PROTECTED]>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > > > We have a hadoop job running and have used custom counters
> > to
> > > > >> track

Best Regards

Jeff Zhang