Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> response time measurement


Copy link to this message
-
Re: response time measurement
yes, you can do that

and actually there some counters implementations,

  public static enum Counter {  
    NUM_FAILED_MAPS,
    NUM_FAILED_REDUCES,
    TOTAL_LAUNCHED_MAPS,
    TOTAL_LAUNCHED_REDUCES,
    OTHER_LOCAL_MAPS,
    DATA_LOCAL_MAPS,
    RACK_LOCAL_MAPS,
    SLOTS_MILLIS_MAPS,
    SLOTS_MILLIS_REDUCES,
    FALLOW_SLOTS_MILLIS_MAPS,
    FALLOW_SLOTS_MILLIS_REDUCES,
    MAPS_RESPONSE_TIME,
    REDUCE_START_TIME,
    REDUCE_RESPONSE_TIME
  }

Best,

--
Nan Zhu
School of Computer Science,
McGill University

On Saturday, 17 November, 2012 at 9:12 AM, Samaneh Shokuhi wrote:

> Hi,
> I checked the JobInProgress.java class, i see as you said there is no
> counter as "Total Job Time" there, but there are two different variables
> ,one called "finishTime" another called "startTime" .I guess if i add
> (finishTime - startTime) to logJobSummary method that may give us the
> "Total Job Time" .Am i right ?
>
> Best,
> Samaneh
>
>
>
>
> On Tue, Nov 13, 2012 at 1:41 AM, Nan Zhu <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > There is no such a counter in default Hadoop implementation
> >
> > you can instrument your own code in JobInProgress.java to calculate it,
> >
> > there is also an implemented counter in JobInProgress, MAP_SLOTS_MILLS,
> > considering both the map tasks occupying multiple slots, but this value
> > also increase when the task is failed,
> >
> > Best,
> >
> > --
> > Nan Zhu
> > School of Computer Science,
> > McGill University
> >
> >
> > On Monday, 12 November, 2012 at 6:14 PM, Samaneh Shokuhi wrote:
> > > Hi,
> > > I need to do some experience on hadoop source code.
> > > i have modified part of it in MapTask.java class and need to compare the
> > > response time for mapper /job before and after modification .Is it
> > > reported somewhere to see that?
> > >
> > >
> > > Cheers,
> > > Samaneh
> > >
> >
> >
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB