Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> use counter to statistics file row number


Copy link to this message
-
Re: use counter to statistics file row number
Hi ,

mapreduce-user seems like a great place for this discussion.  general does not.

Thanks!
E14

On Jan 19, 2011, at 7:33 PM, venkatesh kavuluri wrote:

> (Bcc general@. This is for Hadoop project level discussions. Includingmapreduce
> -user@)
>
> Liu,
>
> If you want the count of number of records in your input data set, the
> map/reduce framework provides a default counter "Map input records". The
> only caution to follow regarding the custom counters is to not exceed 20 per
> application as they are very expensive. Generally counters are used to track
> few important pieces of information.
>
> Thanks,
> Venkatesh Kavuluri
>
> 2011/1/19 lei liu <[EMAIL PROTECTED]>
>
>> I use counter to statistics  file row number in Mapper, example below code:
>>
>> public void map(LongWritable key, Text value,
>>           OutputCollector<Text, IntWritable> output, Reporter reporter)
>>           throws IOException {
>>
>>       try {
>>
>>           reporter.incrCounter("row", "num", 1);
>>
>>       } catch (Throwable e) {
>>           e.printStackTrace();
>>           throw new RuntimeException(e);
>>       }
>>   }
>>
>> Could everyone tell me whether there are any risks do so?
>>
>>
>> Thanks,
>>
>> LiuLei
>>