Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Job Tracker questions


Copy link to this message
-
Re: Job Tracker questions
JobClient also use proxy of JobTracker.
On Mon, Feb 8, 2010 at 11:19 PM, Mark N <[EMAIL PROTECTED]> wrote:

> Did you check the jobClient source code?
>
>
> On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > I look at the source code, it seems the job tracker web ui also use the
> > proxy of JobTracker to get the counter information rather the xml file.
> >
> >
> > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[EMAIL PROTECTED]> wrote:
> >
> > > yes we can create a webservice in java which would be called by .net to
> > > display these counters.
> > >
> > > But since the java code to read these counters needs use hadoop APIs  (
> > job
> > > client  ) ,  am not sure we can create a webservice to read the
> counters
> > >
> > > Question is how does the default hadoop task tracker display counter
> > > information in JSP pages ? does it read from the XML files ?
> > >
> > > thanks,
> > >
> > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > >
> > > > I think you can create web service using Java, and then in .net using
> > the
> > > > web service to display the result.
> > > >
> > > >
> > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Do you mean want to connect the JobTracker using .Net ? If so, I'm
> > > afraid
> > > > I
> > > > > have no idea how to this. The rpc of hadoop is language dependent.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > >> could you please elaborate on this  ( * hint to get started  as am
> > > very
> > > > >> new
> > > > >> to hadoop? )
> > > > >> So far I could succesfully read all the default and custom
> counters.
> > > > >>
> > > > >> Currently we are having a .net client.
> > > > >>
> > > > >> thanks in advance.
> > > > >>
> > > > >>
> > > > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > wrote:
> > > > >>
> > > > >> > Well, you can create a proxy of JobTracker in client side, and
> > then
> > > > you
> > > > >> can
> > > > >> > use the API of JobTracker to get the information of jobs. The
> > Proxy
> > > > take
> > > > >> > the
> > > > >> > responsibility of  communication with the Master Node.  Read the
> > > > source
> > > > >> > code
> > > > >> > of JobClient can help you.
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]>
> > > wrote:
> > > > >> >
> > > > >> > > Ye currently am using jobclient to read these counters.
> > > > >> > >
> > > > >> > > But We are not able to use *webservices *because the jar which
> > is
> > > > used
> > > > >> to
> > > > >> > > read the counters from  running hadoop job  is itself a Hadoop
> > > > program
> > > > >> > >
> > > > >> > > If we could have pure Java Api which is run without hadoop
> > command
> > > > >> then
> > > > >> > we
> > > > >> > > could return the counter variable into webservices and show in
> > UI.
> > > > >> > >
> > > > >> > > Any help  or technique to show thsese counters in the UI would
> > be
> > > > >> > > appreciated  ( not necessarily using web service )
> > > > >> > >
> > > > >> > >
> > > > >> > > I am using webservices because I am having .net VB client
> > > > >> > >
> > > > >> > > thanks
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >> > >
> > > > >> > > > I think you can use JobClient to get the counters in your
> web
> > > > >> service.
> > > > >> > > > If you look at the shell script bin/hadoop, you will find
> that
> > > > >> actually
> > > > >> > > > this
> > > > >> > > > shell use the JobClient to get the counters.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <
> [EMAIL PROTECTED]>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > > > We have a hadoop job running and have used custom counters
> > to
> > > > >> track

Best Regards

Jeff Zhang
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB