Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - why reduce doesn't work by HBase?

Copy link to this message
why reduce doesn't work by HBase?
yonghu 2012-11-15, 19:42
Dear all,

I use HBase as data source and hdfs as data sink. I wrote the program
which calculate the versions for each cell as follows:

public class ReturnMore { //only Map side has no problem, has already
returned 3 versions

   public static class Map extends TableMapper<Text, Text>{
      private Text outkey = new Text();
      private Text outval = new Text();
      public void map(ImmutableBytesWritable key, Result values, Context
         KeyValue[] kvs = values.raw();
         String row_key;
         String col_fam;
         String col;
         String val;
         String finalKey;
         String finalValue;
         for(KeyValue kv : kvs){
            row_key = Bytes.toString(kv.getRow());
            col_fam = Bytes.toString(kv.getFamily());
            col = Bytes.toString(kv.getQualifier());
            val = Bytes.toString(kv.getValue());
            finalKey = row_key + "/" + col_fam + "/" + col;
            finalValue = new Long(kv.getTimestamp()).toString() + "/" +
            try {
               context.write(outkey, outval);
            } catch (IOException e) {
            } catch (InterruptedException e) {

   public static class Reduce extends Reducer<Text, Text, Text,
IntWritable>{ //the desired output is key + 3
      private IntWritable occ = new IntWritable();
      public void reduce(Text key, Iterator<Text> values, Context
         int i = 0;
         try {
            context.write(key, occ);
         } catch (IOException e) {
         } catch (InterruptedException e) {
   public static void main(String[] args) throws Exception{
      Configuration conf = HBaseConfiguration.create();
      Job job = new Job(conf);
      Scan scan = new Scan();
      TableMapReduceUtil.initTableMapperJob("test", scan, Map.class,
Text.class, Text.class, job);
      FileSystem file = FileSystem.get(conf);
      Path path = new Path("/hans/test/");
      FileOutputFormat.setOutputPath(job, path);

I tested this code in both hbase 0.92.1 and 0.94. However, if I run
this code, it always outputs the content for each cell not as the
output as I defined in reduce function (key + occurrence for each
cell). Can anyone give me advices? By the way, I run it on