Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> how to increment counters inside of InputFormat/RecordReader in mapreduce api?


+
Jim Donofrio 2012-07-30, 06:11
Copy link to this message
-
Re: how to increment counters inside of InputFormat/RecordReader in mapreduce api?
Jim,

This is fixed in 2.x releases already via
https://issues.apache.org/jira/browse/MAPREDUCE-1905 (incompat
change). In 1.x this is a known limitation. Perhaps we can come up
with a different non-breaking fix for 1.x?

On Mon, Jul 30, 2012 at 11:41 AM, Jim Donofrio <[EMAIL PROTECTED]> wrote:
> In the mapred api getRecordReader was passed a Reporter which could then get
> passed to the RecordReader to allow a RecordReader to increment counters for
> different types of records, bad records, etc. In the new mapreduce api
> createRecordReader only gets the InputSplit and TaskAttemptContext, both
> have no access to counters.
>
> Is there really no way to increment counters inside of a RecordReader or
> InputFormat in the mapreduce api?

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB