Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Job Tracker questions


Copy link to this message
-
Re: Job Tracker questions
I think you can create web service using Java, and then in .net using the
web service to display the result.
On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid I
> have no idea how to this. The rpc of hadoop is language dependent.
>
>
>
>
> On Thu, Feb 4, 2010 at 7:18 PM, Mark N <[EMAIL PROTECTED]> wrote:
>
>> could you please elaborate on this  ( * hint to get started  as am very
>> new
>> to hadoop? )
>> So far I could succesfully read all the default and custom counters.
>>
>> Currently we are having a .net client.
>>
>> thanks in advance.
>>
>>
>> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>>
>> > Well, you can create a proxy of JobTracker in client side, and then you
>> can
>> > use the API of JobTracker to get the information of jobs. The Proxy take
>> > the
>> > responsibility of  communication with the Master Node.  Read the source
>> > code
>> > of JobClient can help you.
>> >
>> >
>> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <[EMAIL PROTECTED]> wrote:
>> >
>> > > Ye currently am using jobclient to read these counters.
>> > >
>> > > But We are not able to use *webservices *because the jar which is used
>> to
>> > > read the counters from  running hadoop job  is itself a Hadoop program
>> > >
>> > > If we could have pure Java Api which is run without hadoop command
>> then
>> > we
>> > > could return the counter variable into webservices and show in UI.
>> > >
>> > > Any help  or technique to show thsese counters in the UI would be
>> > > appreciated  ( not necessarily using web service )
>> > >
>> > >
>> > > I am using webservices because I am having .net VB client
>> > >
>> > > thanks
>> > >
>> > >
>> > >
>> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>> > >
>> > > > I think you can use JobClient to get the counters in your web
>> service.
>> > > > If you look at the shell script bin/hadoop, you will find that
>> actually
>> > > > this
>> > > > shell use the JobClient to get the counters.
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]>
>> wrote:
>> > > >
>> > > > > We have a hadoop job running and have used custom counters to
>> track
>> > >  few
>> > > > > counters ( like no of successfully processed documents matching
>> > certain
>> > > > > conditions)
>> > > > >
>> > > > >
>> > > > > Since we need to get this counters even while the Hadoop job is
>> > running
>> > > ,
>> > > > > we
>> > > > > wrote another Java program to read these counters
>> > > > >
>> > > > >
>> > > > > *Counter reader  program *will do the following :
>> > > > >
>> > > > >
>> > > > > 1)      List all the running jobs.
>> > > > >
>> > > > > 2)   Get the running job using Job name
>> > > > >
>> > > > > 2)     Get all the counter for individual running jobs
>> > > > >
>> > > > > 3)      Set this counters in variables.
>> > > > >        We could successfully read these counters  , but since we
>> need
>> > > to
>> > > > > show these counters to custom UI , how can we show these counters?
>> > > > >
>> > > > >        we looked into various options to read these counters to
>> show
>> > in
>> > > > UI
>> > > > > as following :
>> > > > >
>> > > > >      1. Dump these counters to database , however this may be
>> > overhead
>> > > > >      2. Write web service   and UI will invoke the functions from
>> > these
>> > > > > service to show in UI ( However since we need to run "*Counter
>> reader
>> > > > > program "  *with Hadoop command it might not be feasible to write
>> web
>> > > > > service ?   )
>> > > > >
>> > > > >      so the question is can we achive to read the counters using
>> > simple
>> > > > > Java APIs ? Does anyone have idea how does the default jobtracker
>> JSP
>> > > > works
>> > > > > ? we wanted to built something similar to this
>> > > > >
>> > > > > thanks
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Nipen Mark

Best Regards

Jeff Zhang
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB