Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Job Tracker questions


Copy link to this message
-
Re: Job Tracker questions
I think you can create web service using Java, and then in .net using the
web service to display the result.
On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid I
> have no idea how to this. The rpc of hadoop is language dependent.
>
>
>
>
> On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]> wrote:
>
>> could you please elaborate on this  ( * hint to get started  as am very
>> new
>> to hadoop? )
>> So far I could succesfully read all the default and custom counters.
>>
>> Currently we are having a .net client.
>>
>> thanks in advance.
>>
>>
>> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>>
>> > Well, you can create a proxy of JobTracker in client side, and then you
>> can
>> > use the API of JobTracker to get the information of jobs. The Proxy take
>> > the
>> > responsibility of  communication with the Master Node.  Read the source
>> > code
>> > of JobClient can help you.
>> >
>> >
>> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]> wrote:
>> >
>> > > Ye currently am using jobclient to read these counters.
>> > >
>> > > But We are not able to use *webservices *because the jar which is used
>> to
>> > > read the counters from  running hadoop job  is itself a Hadoop program
>> > >
>> > > If we could have pure Java Api which is run without hadoop command
>> then
>> > we
>> > > could return the counter variable into webservices and show in UI.
>> > >
>> > > Any help  or technique to show thsese counters in the UI would be
>> > > appreciated  ( not necessarily using web service )
>> > >
>> > >
>> > > I am using webservices because I am having .net VB client
>> > >
>> > > thanks
>> > >
>> > >
>> > >
>> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>> > >
>> > > > I think you can use JobClient to get the counters in your web
>> service.
>> > > > If you look at the shell script bin/hadoop, you will find that
>> actually
>> > > > this
>> > > > shell use the JobClient to get the counters.
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]>
>> wrote:
>> > > >
>> > > > > We have a hadoop job running and have used custom counters to
>> track
>> > >  few
>> > > > > counters ( like no of successfully processed documents matching
>> > certain
>> > > > > conditions)
>> > > > >
>> > > > >
>> > > > > Since we need to get this counters even while the Hadoop job is
>> > running
>> > > ,
>> > > > > we
>> > > > > wrote another Java program to read these counters
>> > > > >
>> > > > >
>> > > > > *Counter reader  program *will do the following :
>> > > > >
>> > > > >
>> > > > > 1)      List all the running jobs.
>> > > > >
>> > > > > 2)   Get the running job using Job name
>> > > > >
>> > > > > 2)     Get all the counter for individual running jobs
>> > > > >
>> > > > > 3)      Set this counters in variables.
>> > > > >        We could successfully read these counters  , but since we
>> need
>> > > to
>> > > > > show these counters to custom UI , how can we show these counters?
>> > > > >
>> > > > >        we looked into various options to read these counters to
>> show
>> > in
>> > > > UI
>> > > > > as following :
>> > > > >
>> > > > >      1. Dump these counters to database , however this may be
>> > overhead
>> > > > >      2. Write web service   and UI will invoke the functions from
>> > these
>> > > > > service to show in UI ( However since we need to run "*Counter
>> reader
>> > > > > program "  *with Hadoop command it might not be feasible to write
>> web
>> > > > > service ?   )
>> > > > >
>> > > > >      so the question is can we achive to read the counters using
>> > simple
>> > > > > Java APIs ? Does anyone have idea how does the default jobtracker
>> JSP
>> > > > works
>> > > > > ? we wanted to built something similar to this
>> > > > >
>> > > > > thanks
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Nipen Mark

Best Regards

Jeff Zhang