Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> use counter to statistics file row number


Copy link to this message
-
Re: use counter to statistics file row number
Hi ,

mapreduce-user seems like a great place for this discussion.  general does not.

Thanks!
E14

On Jan 19, 2011, at 7:33 PM, venkatesh kavuluri wrote:

> (Bcc general@. This is for Hadoop project level discussions. Includingmapreduce
> -user@)
>
> Liu,
>
> If you want the count of number of records in your input data set, the
> map/reduce framework provides a default counter "Map input records". The
> only caution to follow regarding the custom counters is to not exceed 20 per
> application as they are very expensive. Generally counters are used to track
> few important pieces of information.
>
> Thanks,
> Venkatesh Kavuluri
>
> 2011/1/19 lei liu <[EMAIL PROTECTED]>
>
>> I use counter to statistics  file row number in Mapper, example below code:
>>
>> public void map(LongWritable key, Text value,
>>           OutputCollector<Text, IntWritable> output, Reporter reporter)
>>           throws IOException {
>>
>>       try {
>>
>>           reporter.incrCounter("row", "num", 1);
>>
>>       } catch (Throwable e) {
>>           e.printStackTrace();
>>           throw new RuntimeException(e);
>>       }
>>   }
>>
>> Could everyone tell me whether there are any risks do so?
>>
>>
>> Thanks,
>>
>> LiuLei
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB