Well, you can create a proxy of JobTracker in client side, and then you can
use the API of JobTracker to get the information of jobs. The Proxy take the
responsibility of  communication with the Master Node.  Read the source code
of JobClient can help you.
On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]> wrote:

> Ye currently am using jobclient to read these counters.
>
> But We are not able to use *webservices *because the jar which is used to
> read the counters from  running hadoop job  is itself a Hadoop program
>
> If we could have pure Java Api which is run without hadoop command then we
> could return the counter variable into webservices and show in UI.
>
> Any help  or technique to show thsese counters in the UI would be
> appreciated  ( not necessarily using web service )
>
>
> I am using webservices because I am having .net VB client
>
> thanks
>
>
>
> On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > I think you can use JobClient to get the counters in your web service.
> > If you look at the shell script bin/hadoop, you will find that actually
> > this
> > shell use the JobClient to get the counters.
> >
> >
> >
> > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]> wrote:
> >
> > > We have a hadoop job running and have used custom counters to track
>  few
> > > counters ( like no of successfully processed documents matching certain
> > > conditions)
> > >
> > >
> > > Since we need to get this counters even while the Hadoop job is running
> ,
> > > we
> > > wrote another Java program to read these counters
> > >
> > >
> > > *Counter reader  program *will do the following :
> > >
> > >
> > > 1)      List all the running jobs.
> > >
> > > 2)   Get the running job using Job name
> > >
> > > 2)     Get all the counter for individual running jobs
> > >
> > > 3)      Set this counters in variables.
> > >        We could successfully read these counters  , but since we need
> to
> > > show these counters to custom UI , how can we show these counters?
> > >
> > >        we looked into various options to read these counters to show in
> > UI
> > > as following :
> > >
> > >      1. Dump these counters to database , however this may be overhead
> > >      2. Write web service   and UI will invoke the functions from these
> > > service to show in UI ( However since we need to run "*Counter reader
> > > program "  *with Hadoop command it might not be feasible to write web
> > > service ?   )
> > >
> > >      so the question is can we achive to read the counters using simple
> > > Java APIs ? Does anyone have idea how does the default jobtracker JSP
> > works
> > > ? we wanted to built something similar to this
> > >
> > > thanks
> > >
> > >
> > >
> > > --
> > > Nipen Mark
> > >
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>
>
>
> --
> Nipen Mark
>

--
Best Regards

Jeff Zhang

could you please elaborate on this  ( * hint to get started  as am very new
to hadoop? )
So far I could succesfully read all the default and custom counters.

Currently we are having a .net client.

thanks in advance.
On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> Well, you can create a proxy of JobTracker in client side, and then you can
> use the API of JobTracker to get the information of jobs. The Proxy take
> the
> responsibility of  communication with the Master Node.  Read the source
> code
> of JobClient can help you.
>
>
> On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]> wrote:
>
> > Ye currently am using jobclient to read these counters.
> >
> > But We are not able to use *webservices *because the jar which is used to
> > read the counters from  running hadoop job  is itself a Hadoop program
> >
> > If we could have pure Java Api which is run without hadoop command then
> we
> > could return the counter variable into webservices and show in UI.
> >
> > Any help  or technique to show thsese counters in the UI would be
> > appreciated  ( not necessarily using web service )
> >
> >
> > I am using webservices because I am having .net VB client
> >
> > thanks
> >
> >
> >
> > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> >
> > > I think you can use JobClient to get the counters in your web service.
> > > If you look at the shell script bin/hadoop, you will find that actually
> > > this
> > > shell use the JobClient to get the counters.
> > >
> > >
> > >
> > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]> wrote:
> > >
> > > > We have a hadoop job running and have used custom counters to track
> >  few
> > > > counters ( like no of successfully processed documents matching
> certain
> > > > conditions)
> > > >
> > > >
> > > > Since we need to get this counters even while the Hadoop job is
> running
> > ,
> > > > we
> > > > wrote another Java program to read these counters
> > > >
> > > >
> > > > *Counter reader  program *will do the following :
> > > >
> > > >
> > > > 1)      List all the running jobs.
> > > >
> > > > 2)   Get the running job using Job name
> > > >
> > > > 2)     Get all the counter for individual running jobs
> > > >
> > > > 3)      Set this counters in variables.
> > > >        We could successfully read these counters  , but since we need
> > to
> > > > show these counters to custom UI , how can we show these counters?
> > > >
> > > >        we looked into various options to read these counters to show
> in
> > > UI
> > > > as following :
> > > >
> > > >      1. Dump these counters to database , however this may be
> overhead
> > > >      2. Write web service   and UI will invoke the functions from
> these
> > > > service to show in UI ( However since we need to run "*Counter reader
> > > > program "  *with Hadoop command it might not be feasible to write web
> > > > service ?   )
> > > >
> > > >      so the question is can we achive to read the counters using
> simple
> > > > Java APIs ? Does anyone have idea how does the default jobtracker JSP
> > > works
> > > > ? we wanted to built something similar to this
> > > >
> > > > thanks
> > > >
> > > >
> > > >
> > > > --
> > > > Nipen Mark
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> >
> >
> >
> > --
> > Nipen Mark
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

--
Nipen Mark

Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid I
have no idea how to this. The rpc of hadoop is language dependent.

On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]> wrote:

> could you please elaborate on this  ( * hint to get started  as am very new
> to hadoop? )
> So far I could succesfully read all the default and custom counters.
>
> Currently we are having a .net client.
>
> thanks in advance.
>
>
> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > Well, you can create a proxy of JobTracker in client side, and then you
> can
> > use the API of JobTracker to get the information of jobs. The Proxy take
> > the
> > responsibility of  communication with the Master Node.  Read the source
> > code
> > of JobClient can help you.
> >
> >
> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]> wrote:
> >
> > > Ye currently am using jobclient to read these counters.
> > >
> > > But We are not able to use *webservices *because the jar which is used
> to
> > > read the counters from  running hadoop job  is itself a Hadoop program
> > >
> > > If we could have pure Java Api which is run without hadoop command then
> > we
> > > could return the counter variable into webservices and show in UI.
> > >
> > > Any help  or technique to show thsese counters in the UI would be
> > > appreciated  ( not necessarily using web service )
> > >
> > >
> > > I am using webservices because I am having .net VB client
> > >
> > > thanks
> > >
> > >
> > >
> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > >
> > > > I think you can use JobClient to get the counters in your web
> service.
> > > > If you look at the shell script bin/hadoop, you will find that
> actually
> > > > this
> > > > shell use the JobClient to get the counters.
> > > >
> > > >
> > > >
> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > We have a hadoop job running and have used custom counters to track
> > >  few
> > > > > counters ( like no of successfully processed documents matching
> > certain
> > > > > conditions)
> > > > >
> > > > >
> > > > > Since we need to get this counters even while the Hadoop job is
> > running
> > > ,
> > > > > we
> > > > > wrote another Java program to read these counters
> > > > >
> > > > >
> > > > > *Counter reader  program *will do the following :
> > > > >
> > > > >
> > > > > 1)      List all the running jobs.
> > > > >
> > > > > 2)   Get the running job using Job name
> > > > >
> > > > > 2)     Get all the counter for individual running jobs
> > > > >
> > > > > 3)      Set this counters in variables.
> > > > >        We could successfully read these counters  , but since we
> need
> > > to
> > > > > show these counters to custom UI , how can we show these counters?
> > > > >
> > > > >        we looked into various options to read these counters to
> show
> > in
> > > > UI
> > > > > as following :
> > > > >
> > > > >      1. Dump these counters to database , however this may be
> > overhead
> > > > >      2. Write web service   and UI will invoke the functions from
> > these
> > > > > service to show in UI ( However since we need to run "*Counter
> reader
> > > > > program "  *with Hadoop command it might not be feasible to write
> web
> > > > > service ?   )
> > > > >
> > > > >      so the question is can we achive to read the counters using
> > simple
> > > > > Java APIs ? Does anyone have idea how does the default jobtracker
> JSP
> > > > works
> > > > > ? we wanted to built something similar to this
> > > > >
> > > > > thanks
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Nipen Mark
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards
> > > >
> > > > Jeff Zhang
> > > >
> > >
> > >
> > >
> > > --
> > > Nipen Mark
> > >
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>
>
>
> --
> Nipen Mark
>

--
Best Regards

Jeff Zhang

I think you can create web service using Java, and then in .net using the
web service to display the result.
On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid I
> have no idea how to this. The rpc of hadoop is language dependent.
>
>
>
>
> On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]> wrote:
>
>> could you please elaborate on this  ( * hint to get started  as am very
>> new
>> to hadoop? )
>> So far I could succesfully read all the default and custom counters.
>>
>> Currently we are having a .net client.
>>
>> thanks in advance.
>>
>>
>> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>>
>> > Well, you can create a proxy of JobTracker in client side, and then you
>> can
>> > use the API of JobTracker to get the information of jobs. The Proxy take
>> > the
>> > responsibility of  communication with the Master Node.  Read the source
>> > code
>> > of JobClient can help you.
>> >
>> >
>> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]> wrote:
>> >
>> > > Ye currently am using jobclient to read these counters.
>> > >
>> > > But We are not able to use *webservices *because the jar which is used
>> to
>> > > read the counters from  running hadoop job  is itself a Hadoop program
>> > >
>> > > If we could have pure Java Api which is run without hadoop command
>> then
>> > we
>> > > could return the counter variable into webservices and show in UI.
>> > >
>> > > Any help  or technique to show thsese counters in the UI would be
>> > > appreciated  ( not necessarily using web service )
>> > >
>> > >
>> > > I am using webservices because I am having .net VB client
>> > >
>> > > thanks
>> > >
>> > >
>> > >
>> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>> > >
>> > > > I think you can use JobClient to get the counters in your web
>> service.
>> > > > If you look at the shell script bin/hadoop, you will find that
>> actually
>> > > > this
>> > > > shell use the JobClient to get the counters.
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]>
>> wrote:
>> > > >
>> > > > > We have a hadoop job running and have used custom counters to
>> track
>> > >  few
>> > > > > counters ( like no of successfully processed documents matching
>> > certain
>> > > > > conditions)
>> > > > >
>> > > > >
>> > > > > Since we need to get this counters even while the Hadoop job is
>> > running
>> > > ,
>> > > > > we
>> > > > > wrote another Java program to read these counters
>> > > > >
>> > > > >
>> > > > > *Counter reader  program *will do the following :
>> > > > >
>> > > > >
>> > > > > 1)      List all the running jobs.
>> > > > >
>> > > > > 2)   Get the running job using Job name
>> > > > >
>> > > > > 2)     Get all the counter for individual running jobs
>> > > > >
>> > > > > 3)      Set this counters in variables.
>> > > > >        We could successfully read these counters  , but since we
>> need
>> > > to
>> > > > > show these counters to custom UI , how can we show these counters?
>> > > > >
>> > > > >        we looked into various options to read these counters to
>> show
>> > in
>> > > > UI
>> > > > > as following :
>> > > > >
>> > > > >      1. Dump these counters to database , however this may be
>> > overhead
>> > > > >      2. Write web service   and UI will invoke the functions from
>> > these
>> > > > > service to show in UI ( However since we need to run "*Counter
>> reader
>> > > > > program "  *with Hadoop command it might not be feasible to write
>> web
>> > > > > service ?   )
>> > > > >
>> > > > >      so the question is can we achive to read the counters using
>> > simple
>> > > > > Java APIs ? Does anyone have idea how does the default jobtracker
>> JSP
>> > > > works
>> > > > > ? we wanted to built something similar to this
>> > > > >
>> > > > > thanks
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Nipen Mark

Best Regards

Jeff Zhang

You can use org.apache.hadoop.ipc.RPC.getProxy(....) to initialize the proxy
of JobTracker

On Thu, Feb 4, 2010 at 7:23 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> I think you can create web service using Java, and then in .net using the
> web service to display the result.
>
>
>
> On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
>> Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid
>> I have no idea how to this. The rpc of hadoop is language dependent.
>>
>>
>>
>>
>> On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]> wrote:
>>
>>> could you please elaborate on this  ( * hint to get started  as am very
>>> new
>>> to hadoop? )
>>> So far I could succesfully read all the default and custom counters.
>>>
>>> Currently we are having a .net client.
>>>
>>> thanks in advance.
>>>
>>>
>>> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>>>
>>> > Well, you can create a proxy of JobTracker in client side, and then you
>>> can
>>> > use the API of JobTracker to get the information of jobs. The Proxy
>>> take
>>> > the
>>> > responsibility of  communication with the Master Node.  Read the source
>>> > code
>>> > of JobClient can help you.
>>> >
>>> >
>>> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]> wrote:
>>> >
>>> > > Ye currently am using jobclient to read these counters.
>>> > >
>>> > > But We are not able to use *webservices *because the jar which is
>>> used to
>>> > > read the counters from  running hadoop job  is itself a Hadoop
>>> program
>>> > >
>>> > > If we could have pure Java Api which is run without hadoop command
>>> then
>>> > we
>>> > > could return the counter variable into webservices and show in UI.
>>> > >
>>> > > Any help  or technique to show thsese counters in the UI would be
>>> > > appreciated  ( not necessarily using web service )
>>> > >
>>> > >
>>> > > I am using webservices because I am having .net VB client
>>> > >
>>> > > thanks
>>> > >
>>> > >
>>> > >
>>> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>>> > >
>>> > > > I think you can use JobClient to get the counters in your web
>>> service.
>>> > > > If you look at the shell script bin/hadoop, you will find that
>>> actually
>>> > > > this
>>> > > > shell use the JobClient to get the counters.
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]>
>>> wrote:
>>> > > >
>>> > > > > We have a hadoop job running and have used custom counters to
>>> track
>>> > >  few
>>> > > > > counters ( like no of successfully processed documents matching
>>> > certain
>>> > > > > conditions)
>>> > > > >
>>> > > > >
>>> > > > > Since we need to get this counters even while the Hadoop job is
>>> > running
>>> > > ,
>>> > > > > we
>>> > > > > wrote another Java program to read these counters
>>> > > > >
>>> > > > >
>>> > > > > *Counter reader  program *will do the following :
>>> > > > >
>>> > > > >
>>> > > > > 1)      List all the running jobs.
>>> > > > >
>>> > > > > 2)   Get the running job using Job name
>>> > > > >
>>> > > > > 2)     Get all the counter for individual running jobs
>>> > > > >
>>> > > > > 3)      Set this counters in variables.
>>> > > > >        We could successfully read these counters  , but since we
>>> need
>>> > > to
>>> > > > > show these counters to custom UI , how can we show these
>>> counters?
>>> > > > >
>>> > > > >        we looked into various options to read these counters to
>>> show
>>> > in
>>> > > > UI
>>> > > > > as following :
>>> > > > >
>>> > > > >      1. Dump these counters to database , however this may be
>>> > overhead
>>> > > > >      2. Write web service   and UI will invoke the functions from
>>> > these
>>> > > > > service to show in UI ( However since we need to run "*Counter
>>> reader
>>> > > > > program "  *with Hadoop command it might not be feasible to write
>>> web
>>> > > > > service ?   )
>>> > > > >
>>> > > > >      so the question is can we achive to read the counters using

Best Regards

Jeff Zhang

yes we can create a webservice in java which would be called by .net to
display these counters.

But since the java code to read these counters needs use hadoop APIs  ( job
client  ) ,  am not sure we can create a webservice to read the counters

Question is how does the default hadoop task tracker display counter
information in JSP pages ? does it read from the XML files ?

thanks,

On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> I think you can create web service using Java, and then in .net using the
> web service to display the result.
>
>
> On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid
> I
> > have no idea how to this. The rpc of hadoop is language dependent.
> >
> >
> >
> >
> > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]> wrote:
> >
> >> could you please elaborate on this  ( * hint to get started  as am very
> >> new
> >> to hadoop? )
> >> So far I could succesfully read all the default and custom counters.
> >>
> >> Currently we are having a .net client.
> >>
> >> thanks in advance.
> >>
> >>
> >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> >>
> >> > Well, you can create a proxy of JobTracker in client side, and then
> you
> >> can
> >> > use the API of JobTracker to get the information of jobs. The Proxy
> take
> >> > the
> >> > responsibility of  communication with the Master Node.  Read the
> source
> >> > code
> >> > of JobClient can help you.
> >> >
> >> >
> >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > Ye currently am using jobclient to read these counters.
> >> > >
> >> > > But We are not able to use *webservices *because the jar which is
> used
> >> to
> >> > > read the counters from  running hadoop job  is itself a Hadoop
> program
> >> > >
> >> > > If we could have pure Java Api which is run without hadoop command
> >> then
> >> > we
> >> > > could return the counter variable into webservices and show in UI.
> >> > >
> >> > > Any help  or technique to show thsese counters in the UI would be
> >> > > appreciated  ( not necessarily using web service )
> >> > >
> >> > >
> >> > > I am using webservices because I am having .net VB client
> >> > >
> >> > > thanks
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]>
> wrote:
> >> > >
> >> > > > I think you can use JobClient to get the counters in your web
> >> service.
> >> > > > If you look at the shell script bin/hadoop, you will find that
> >> actually
> >> > > > this
> >> > > > shell use the JobClient to get the counters.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]>
> >> wrote:
> >> > > >
> >> > > > > We have a hadoop job running and have used custom counters to
> >> track
> >> > >  few
> >> > > > > counters ( like no of successfully processed documents matching
> >> > certain
> >> > > > > conditions)
> >> > > > >
> >> > > > >
> >> > > > > Since we need to get this counters even while the Hadoop job is
> >> > running
> >> > > ,
> >> > > > > we
> >> > > > > wrote another Java program to read these counters
> >> > > > >
> >> > > > >
> >> > > > > *Counter reader  program *will do the following :
> >> > > > >
> >> > > > >
> >> > > > > 1)      List all the running jobs.
> >> > > > >
> >> > > > > 2)   Get the running job using Job name
> >> > > > >
> >> > > > > 2)     Get all the counter for individual running jobs
> >> > > > >
> >> > > > > 3)      Set this counters in variables.
> >> > > > >        We could successfully read these counters  , but since we
> >> need
> >> > > to
> >> > > > > show these counters to custom UI , how can we show these
> counters?
> >> > > > >
> >> > > > >        we looked into various options to read these counters to
> >> show
> >> > in
> >> > > > UI
> >> > > > > as following :
> >> > > > >
> >> > > > >      1. Dump these counters to database , however this may be

Nipen Mark

I look at the source code, it seems the job tracker web ui also use the
proxy of JobTracker to get the counter information rather the xml file.
On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[EMAIL PROTECTED]> wrote:

> yes we can create a webservice in java which would be called by .net to
> display these counters.
>
> But since the java code to read these counters needs use hadoop APIs  ( job
> client  ) ,  am not sure we can create a webservice to read the counters
>
> Question is how does the default hadoop task tracker display counter
> information in JSP pages ? does it read from the XML files ?
>
> thanks,
>
> On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > I think you can create web service using Java, and then in .net using the
> > web service to display the result.
> >
> >
> > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> >
> > > Do you mean want to connect the JobTracker using .Net ? If so, I'm
> afraid
> > I
> > > have no idea how to this. The rpc of hadoop is language dependent.
> > >
> > >
> > >
> > >
> > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]> wrote:
> > >
> > >> could you please elaborate on this  ( * hint to get started  as am
> very
> > >> new
> > >> to hadoop? )
> > >> So far I could succesfully read all the default and custom counters.
> > >>
> > >> Currently we are having a .net client.
> > >>
> > >> thanks in advance.
> > >>
> > >>
> > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > >>
> > >> > Well, you can create a proxy of JobTracker in client side, and then
> > you
> > >> can
> > >> > use the API of JobTracker to get the information of jobs. The Proxy
> > take
> > >> > the
> > >> > responsibility of  communication with the Master Node.  Read the
> > source
> > >> > code
> > >> > of JobClient can help you.
> > >> >
> > >> >
> > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]>
> wrote:
> > >> >
> > >> > > Ye currently am using jobclient to read these counters.
> > >> > >
> > >> > > But We are not able to use *webservices *because the jar which is
> > used
> > >> to
> > >> > > read the counters from  running hadoop job  is itself a Hadoop
> > program
> > >> > >
> > >> > > If we could have pure Java Api which is run without hadoop command
> > >> then
> > >> > we
> > >> > > could return the counter variable into webservices and show in UI.
> > >> > >
> > >> > > Any help  or technique to show thsese counters in the UI would be
> > >> > > appreciated  ( not necessarily using web service )
> > >> > >
> > >> > >
> > >> > > I am using webservices because I am having .net VB client
> > >> > >
> > >> > > thanks
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > wrote:
> > >> > >
> > >> > > > I think you can use JobClient to get the counters in your web
> > >> service.
> > >> > > > If you look at the shell script bin/hadoop, you will find that
> > >> actually
> > >> > > > this
> > >> > > > shell use the JobClient to get the counters.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]>
> > >> wrote:
> > >> > > >
> > >> > > > > We have a hadoop job running and have used custom counters to
> > >> track
> > >> > >  few
> > >> > > > > counters ( like no of successfully processed documents
> matching
> > >> > certain
> > >> > > > > conditions)
> > >> > > > >
> > >> > > > >
> > >> > > > > Since we need to get this counters even while the Hadoop job
> is
> > >> > running
> > >> > > ,
> > >> > > > > we
> > >> > > > > wrote another Java program to read these counters
> > >> > > > >
> > >> > > > >
> > >> > > > > *Counter reader  program *will do the following :
> > >> > > > >
> > >> > > > >
> > >> > > > > 1)      List all the running jobs.
> > >> > > > >
> > >> > > > > 2)   Get the running job using Job name
> > >> > > > >
> > >> > > > > 2)     Get all the counter for individual running jobs

Best Regards

Jeff Zhang

Did you check the jobClient source code?
On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> I look at the source code, it seems the job tracker web ui also use the
> proxy of JobTracker to get the counter information rather the xml file.
>
>
> On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[EMAIL PROTECTED]> wrote:
>
> > yes we can create a webservice in java which would be called by .net to
> > display these counters.
> >
> > But since the java code to read these counters needs use hadoop APIs  (
> job
> > client  ) ,  am not sure we can create a webservice to read the counters
> >
> > Question is how does the default hadoop task tracker display counter
> > information in JSP pages ? does it read from the XML files ?
> >
> > thanks,
> >
> > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> >
> > > I think you can create web service using Java, and then in .net using
> the
> > > web service to display the result.
> > >
> > >
> > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > >
> > > > Do you mean want to connect the JobTracker using .Net ? If so, I'm
> > afraid
> > > I
> > > > have no idea how to this. The rpc of hadoop is language dependent.
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]> wrote:
> > > >
> > > >> could you please elaborate on this  ( * hint to get started  as am
> > very
> > > >> new
> > > >> to hadoop? )
> > > >> So far I could succesfully read all the default and custom counters.
> > > >>
> > > >> Currently we are having a .net client.
> > > >>
> > > >> thanks in advance.
> > > >>
> > > >>
> > > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]>
> wrote:
> > > >>
> > > >> > Well, you can create a proxy of JobTracker in client side, and
> then
> > > you
> > > >> can
> > > >> > use the API of JobTracker to get the information of jobs. The
> Proxy
> > > take
> > > >> > the
> > > >> > responsibility of  communication with the Master Node.  Read the
> > > source
> > > >> > code
> > > >> > of JobClient can help you.
> > > >> >
> > > >> >
> > > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]>
> > wrote:
> > > >> >
> > > >> > > Ye currently am using jobclient to read these counters.
> > > >> > >
> > > >> > > But We are not able to use *webservices *because the jar which
> is
> > > used
> > > >> to
> > > >> > > read the counters from  running hadoop job  is itself a Hadoop
> > > program
> > > >> > >
> > > >> > > If we could have pure Java Api which is run without hadoop
> command
> > > >> then
> > > >> > we
> > > >> > > could return the counter variable into webservices and show in
> UI.
> > > >> > >
> > > >> > > Any help  or technique to show thsese counters in the UI would
> be
> > > >> > > appreciated  ( not necessarily using web service )
> > > >> > >
> > > >> > >
> > > >> > > I am using webservices because I am having .net VB client
> > > >> > >
> > > >> > > thanks
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > > wrote:
> > > >> > >
> > > >> > > > I think you can use JobClient to get the counters in your web
> > > >> service.
> > > >> > > > If you look at the shell script bin/hadoop, you will find that
> > > >> actually
> > > >> > > > this
> > > >> > > > shell use the JobClient to get the counters.
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > We have a hadoop job running and have used custom counters
> to
> > > >> track
> > > >> > >  few
> > > >> > > > > counters ( like no of successfully processed documents
> > matching
> > > >> > certain
> > > >> > > > > conditions)
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Since we need to get this counters even while the Hadoop job
> > is
> > > >> > running
> > > >> > > ,
> > > >> > > > > we
> > > >> > > > > wrote another Java program to read these counters

Nipen Mark

JobClient also use proxy of JobTracker.
On Mon, Feb 8, 2010 at 11:19 PM, Mark N <[EMAIL PROTECTED]> wrote:

> Did you check the jobClient source code?
>
>
> On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > I look at the source code, it seems the job tracker web ui also use the
> > proxy of JobTracker to get the counter information rather the xml file.
> >
> >
> > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[EMAIL PROTECTED]> wrote:
> >
> > > yes we can create a webservice in java which would be called by .net to
> > > display these counters.
> > >
> > > But since the java code to read these counters needs use hadoop APIs  (
> > job
> > > client  ) ,  am not sure we can create a webservice to read the
> counters
> > >
> > > Question is how does the default hadoop task tracker display counter
> > > information in JSP pages ? does it read from the XML files ?
> > >
> > > thanks,
> > >
> > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > >
> > > > I think you can create web service using Java, and then in .net using
> > the
> > > > web service to display the result.
> > > >
> > > >
> > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Do you mean want to connect the JobTracker using .Net ? If so, I'm
> > > afraid
> > > > I
> > > > > have no idea how to this. The rpc of hadoop is language dependent.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > >> could you please elaborate on this  ( * hint to get started  as am
> > > very
> > > > >> new
> > > > >> to hadoop? )
> > > > >> So far I could succesfully read all the default and custom
> counters.
> > > > >>
> > > > >> Currently we are having a .net client.
> > > > >>
> > > > >> thanks in advance.
> > > > >>
> > > > >>
> > > > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > wrote:
> > > > >>
> > > > >> > Well, you can create a proxy of JobTracker in client side, and
> > then
> > > > you
> > > > >> can
> > > > >> > use the API of JobTracker to get the information of jobs. The
> > Proxy
> > > > take
> > > > >> > the
> > > > >> > responsibility of  communication with the Master Node.  Read the
> > > > source
> > > > >> > code
> > > > >> > of JobClient can help you.
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]>
> > > wrote:
> > > > >> >
> > > > >> > > Ye currently am using jobclient to read these counters.
> > > > >> > >
> > > > >> > > But We are not able to use *webservices *because the jar which
> > is
> > > > used
> > > > >> to
> > > > >> > > read the counters from  running hadoop job  is itself a Hadoop
> > > > program
> > > > >> > >
> > > > >> > > If we could have pure Java Api which is run without hadoop
> > command
> > > > >> then
> > > > >> > we
> > > > >> > > could return the counter variable into webservices and show in
> > UI.
> > > > >> > >
> > > > >> > > Any help  or technique to show thsese counters in the UI would
> > be
> > > > >> > > appreciated  ( not necessarily using web service )
> > > > >> > >
> > > > >> > >
> > > > >> > > I am using webservices because I am having .net VB client
> > > > >> > >
> > > > >> > > thanks
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >> > >
> > > > >> > > > I think you can use JobClient to get the counters in your
> web
> > > > >> service.
> > > > >> > > > If you look at the shell script bin/hadoop, you will find
> that
> > > > >> actually
> > > > >> > > > this
> > > > >> > > > shell use the JobClient to get the counters.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <
> [EMAIL PROTECTED]>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > > > We have a hadoop job running and have used custom counters
> > to
> > > > >> track

Best Regards

Jeff Zhang

i have to find Counter and i am using inetsocketAddress and connecting
into the jobtracker jsp
with help of it am able to find the counter when it is running on pseudo mode

but as i am trying to run this in cluster . i am not able to get any
counters . SI there any parameters that I need to change in
configuration to read the counter information ?
Thanks in advance
On Tue, Feb 9, 2010 at 8:49 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> JobClient also use proxy of JobTracker.
>
>
> On Mon, Feb 8, 2010 at 11:19 PM, Mark N <[EMAIL PROTECTED]> wrote:
>
> > Did you check the jobClient source code?
> >
> >
> > On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> >
> > > I look at the source code, it seems the job tracker web ui also use the
> > > proxy of JobTracker to get the counter information rather the xml file.
> > >
> > >
> > > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[EMAIL PROTECTED]> wrote:
> > >
> > > > yes we can create a webservice in java which would be called by .net
> to
> > > > display these counters.
> > > >
> > > > But since the java code to read these counters needs use hadoop APIs
>  (
> > > job
> > > > client  ) ,  am not sure we can create a webservice to read the
> > counters
> > > >
> > > > Question is how does the default hadoop task tracker display counter
> > > > information in JSP pages ? does it read from the XML files ?
> > > >
> > > > thanks,
> > > >
> > > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > I think you can create web service using Java, and then in .net
> using
> > > the
> > > > > web service to display the result.
> > > > >
> > > > >
> > > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > > > Do you mean want to connect the JobTracker using .Net ? If so,
> I'm
> > > > afraid
> > > > > I
> > > > > > have no idea how to this. The rpc of hadoop is language
> dependent.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]>
> > wrote:
> > > > > >
> > > > > >> could you please elaborate on this  ( * hint to get started  as
> am
> > > > very
> > > > > >> new
> > > > > >> to hadoop? )
> > > > > >> So far I could succesfully read all the default and custom
> > counters.
> > > > > >>
> > > > > >> Currently we are having a .net client.
> > > > > >>
> > > > > >> thanks in advance.
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > > wrote:
> > > > > >>
> > > > > >> > Well, you can create a proxy of JobTracker in client side, and
> > > then
> > > > > you
> > > > > >> can
> > > > > >> > use the API of JobTracker to get the information of jobs. The
> > > Proxy
> > > > > take
> > > > > >> > the
> > > > > >> > responsibility of  communication with the Master Node.  Read
> the
> > > > > source
> > > > > >> > code
> > > > > >> > of JobClient can help you.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]>
> > > > wrote:
> > > > > >> >
> > > > > >> > > Ye currently am using jobclient to read these counters.
> > > > > >> > >
> > > > > >> > > But We are not able to use *webservices *because the jar
> which
> > > is
> > > > > used
> > > > > >> to
> > > > > >> > > read the counters from  running hadoop job  is itself a
> Hadoop
> > > > > program
> > > > > >> > >
> > > > > >> > > If we could have pure Java Api which is run without hadoop
> > > command
> > > > > >> then
> > > > > >> > we
> > > > > >> > > could return the counter variable into webservices and show
> in
> > > UI.
> > > > > >> > >
> > > > > >> > > Any help  or technique to show thsese counters in the UI
> would
> > > be
> > > > > >> > > appreciated  ( not necessarily using web service )
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > I am using webservices because I am having .net VB client
> > > > > >> > >
> > > > > >> > > thanks
> > > > > >> > >
> > > > > >> > >

Nipen Mark

This is the sample code for get the counters for one specified job ( I have
tested it on my cluster).  What you need to change is the jobtracker address
and jobID. Remember to put this class in package org.apache.hadoop.mapred,
becuase the JobSubmissionProtocol is not public.

package org.apache.hadoop.mapred;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.Iterator;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.ipc.RPC;
import org.apache.hadoop.mapred.Counters.Counter;
import org.apache.hadoop.mapred.Counters.Group;

public class MyServer {

    public static void main(String[] args) throws IOException {
        InetSocketAddress address = new InetSocketAddress("sha-cs-04",
9001);
        JobSubmissionProtocol jobClient = (JobSubmissionProtocol)
RPC.getProxy(
                JobSubmissionProtocol.class,
JobSubmissionProtocol.versionID,
                address, new Configuration());
        JobID jobID=new JobID("201002111947",460);
        Counters counters= jobClient.getJobCounters(jobID);
        Iterator<Group> iter=counters.iterator();
        while(iter.hasNext()){
            Group group=iter.next();
            System.out.println(group.getDisplayName());
            Iterator<Counter> cIter=group.iterator();
            while(cIter.hasNext()){
                Counter counter=cIter.next();

System.out.println("\t"+counter.getName()+":"+counter.getValue());
            }
        }
    }
}

On Tue, Feb 23, 2010 at 9:38 PM, Mark N <[EMAIL PROTECTED]> wrote:

> i have to find Counter and i am using inetsocketAddress and connecting
> into the jobtracker jsp
> with help of it am able to find the counter when it is running on pseudo
> mode
>
> but as i am trying to run this in cluster . i am not able to get any
> counters . SI there any parameters that I need to change in
> configuration to read the counter information ?
>
>
> Thanks in advance
>
>
> On Tue, Feb 9, 2010 at 8:49 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> > JobClient also use proxy of JobTracker.
> >
> >
> > On Mon, Feb 8, 2010 at 11:19 PM, Mark N <[EMAIL PROTECTED]> wrote:
> >
> > > Did you check the jobClient source code?
> > >
> > >
> > > On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> > >
> > > > I look at the source code, it seems the job tracker web ui also use
> the
> > > > proxy of JobTracker to get the counter information rather the xml
> file.
> > > >
> > > >
> > > > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > yes we can create a webservice in java which would be called by
> .net
> > to
> > > > > display these counters.
> > > > >
> > > > > But since the java code to read these counters needs use hadoop
> APIs
> >  (
> > > > job
> > > > > client  ) ,  am not sure we can create a webservice to read the
> > > counters
> > > > >
> > > > > Question is how does the default hadoop task tracker display
> counter
> > > > > information in JSP pages ? does it read from the XML files ?
> > > > >
> > > > > thanks,
> > > > >
> > > > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > > > I think you can create web service using Java, and then in .net
> > using
> > > > the
> > > > > > web service to display the result.
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]>
> > wrote:
> > > > > >
> > > > > > > Do you mean want to connect the JobTracker using .Net ? If so,
> > I'm
> > > > > afraid
> > > > > > I
> > > > > > > have no idea how to this. The rpc of hadoop is language
> > dependent.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]>
> > > wrote:
> > > > > > >
> > > > > > >> could you please elaborate on this  ( * hint to get started
>  as
> > am
> > > > > very
> > > > > > >> new
> > > > > > >> to hadoop? )
> > > > > > >> So far I could succesfully read all the default and custom

Best Regards

Jeff Zhang