Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Job Tracker questions


Copy link to this message
-
Re: Job Tracker questions
This is the sample code for get the counters for one specified job ( I have
tested it on my cluster).  What you need to change is the jobtracker address
and jobID. Remember to put this class in package org.apache.hadoop.mapred,
becuase the JobSubmissionProtocol is not public.

package org.apache.hadoop.mapred;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.Iterator;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.ipc.RPC;
import org.apache.hadoop.mapred.Counters.Counter;
import org.apache.hadoop.mapred.Counters.Group;

public class MyServer {

    public static void main(String[] args) throws IOException {
        InetSocketAddress address = new InetSocketAddress("sha-cs-04",
9001);
        JobSubmissionProtocol jobClient = (JobSubmissionProtocol)
RPC.getProxy(
                JobSubmissionProtocol.class,
JobSubmissionProtocol.versionID,
                address, new Configuration());
        JobID jobID=new JobID("201002111947",460);
        Counters counters= jobClient.getJobCounters(jobID);
        Iterator<Group> iter=counters.iterator();
        while(iter.hasNext()){
            Group group=iter.next();
            System.out.println(group.getDisplayName());
            Iterator<Counter> cIter=group.iterator();
            while(cIter.hasNext()){
                Counter counter=cIter.next();

System.out.println("\t"+counter.getName()+":"+counter.getValue());
            }
        }
    }
}

On Tue, Feb 23, 2010 at 9:38 PM, Mark N <[EMAIL PROTECTED]> wrote:

> i have to find Counter and i am using inetsocketAddress and connecting
> into the jobtracker jsp
> with help of it am able to find the counter when it is running on pseudo
> mode
>
> but as i am trying to run this in cluster . i am not able to get any
> counters . SI there any parameters that I need to change in
> configuration to read the counter information ?
>
>
> Thanks in advance
>
>
> On Tue, Feb 9, 2010 at 8:49 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > JobClient also use proxy of JobTracker.
> >
> >
> > On Mon, Feb 8, 2010 at 11:19 PM, Mark N <[EMAIL PROTECTED]> wrote:
> >
> > > Did you check the jobClient source code?
> > >
> > >
> > > On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > >
> > > > I look at the source code, it seems the job tracker web ui also use
> the
> > > > proxy of JobTracker to get the counter information rather the xml
> file.
> > > >
> > > >
> > > > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > yes we can create a webservice in java which would be called by
> .net
> > to
> > > > > display these counters.
> > > > >
> > > > > But since the java code to read these counters needs use hadoop
> APIs
> >  (
> > > > job
> > > > > client  ) ,  am not sure we can create a webservice to read the
> > > counters
> > > > >
> > > > > Question is how does the default hadoop task tracker display
> counter
> > > > > information in JSP pages ? does it read from the XML files ?
> > > > >
> > > > > thanks,
> > > > >
> > > > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > > > I think you can create web service using Java, and then in .net
> > using
> > > > the
> > > > > > web service to display the result.
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > wrote:
> > > > > >
> > > > > > > Do you mean want to connect the JobTracker using .Net ? If so,
> > I'm
> > > > > afraid
> > > > > > I
> > > > > > > have no idea how to this. The rpc of hadoop is language
> > dependent.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]>
> > > wrote:
> > > > > > >
> > > > > > >> could you please elaborate on this  ( * hint to get started
>  as
> > am
> > > > > very
> > > > > > >> new
> > > > > > >> to hadoop? )
> > > > > > >> So far I could succesfully read all the default and custom

Best Regards

Jeff Zhang