Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> Job Tracker questions


Copy link to this message
-
Re: Job Tracker questions
I think you can use JobClient to get the counters in your web service.
If you look at the shell script bin/hadoop, you will find that actually this
shell use the JobClient to get the counters.

On Wed, Feb 3, 2010 at 4:34 AM, Mark N <[EMAIL PROTECTED]> wrote:

> We have a hadoop job running and have used custom counters to track  few
> counters ( like no of successfully processed documents matching certain
> conditions)
>
>
> Since we need to get this counters even while the Hadoop job is running ,
> we
> wrote another Java program to read these counters
>
>
> *Counter reader  program *will do the following :
>
>
> 1)      List all the running jobs.
>
> 2)   Get the running job using Job name
>
> 2)     Get all the counter for individual running jobs
>
> 3)      Set this counters in variables.
>        We could successfully read these counters  , but since we need to
> show these counters to custom UI , how can we show these counters?
>
>        we looked into various options to read these counters to show in UI
> as following :
>
>      1. Dump these counters to database , however this may be overhead
>      2. Write web service   and UI will invoke the functions from these
> service to show in UI ( However since we need to run "*Counter reader
> program "  *with Hadoop command it might not be feasible to write web
> service ?   )
>
>      so the question is can we achive to read the counters using simple
> Java APIs ? Does anyone have idea how does the default jobtracker JSP works
> ? we wanted to built something similar to this
>
> thanks
>
>
>
> --
> Nipen Mark
>

--
Best Regards

Jeff Zhang