Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo, mail # dev - Combiner Iterator


Copy link to this message
-
Re: Combiner Iterator
Miguel Pereira 2013-09-18, 13:55
So, the value iterator is in the abstract class combiner. The super class
of the statscombiner. It automatically passes in the valueiterator to the
reduce method.

in the findTop() method

>         Iterator<Value> viter = new ValueIterator(getSource());
>         topValue = reduce(topKey, viter);
>

this is accomplished by loosening the equality check on the row to exclude
timestamp. meaning rows with similar keys minus the timestamp will be
considered equal.

So, I think, all you need to do is attach your combiner to the table /
column family.
Have you run the statscombiner example on the shell ?

Cheers,
Miguel
On Wed, Sep 18, 2013 at 9:23 AM, Devin Pinkston <[EMAIL PROTECTED]>wrote:

> Hello,
>
> I am trying to work with the example combiner iterator through java code
> instead of the jar or shell.  My question is how do I pass in the
> Iterator<Value> to the reduce method?  Usually I would create a Key
> Value Iterator, but this requires an Iterator just over the Value, and
> then the key to be passed in separately.  The reducer method comes from
> the StatsCombiner class under examples/simple/combiner.  What I have
> right now:
>
>  Iterator<Map.Entry<Key, Value>> iterator = scan.iterator();
>         Iterator<Value> iter;
>
>
>         while (iterator.hasNext()) {
>             Map.Entry<Key, Value> entry = iterator.next();
>             iter = iterator.next().getValue();
>             Key key = entry.getKey();
>             Value value = entry.getValue();
>             reduce(key, iter);
>         }
>
> How would I create the Iterator<Value>?  Every way I have tried has
> led me to dead end or error at runtime.
>
> Thanks!
>
> Here is the reduce method from StatsCombiner.java:
>
>  @Override
>   public Value reduce(Key key, Iterator<Value> iter) {
>
>     long min = Long.MAX_VALUE;
>     long max = Long.MIN_VALUE;
>     long sum = 0;
>     long count = 0;
>
>     while (iter.hasNext()) {
>       String stats[] = iter.next().toString().split(",");
>
>       if (stats.length == 1) {
>         long val = Long.parseLong(stats[0], radix);
>         min = Math.min(val, min);
>         max = Math.max(val, max);
>         sum += val;
>         count += 1;
>       } else {
>         min = Math.min(Long.parseLong(stats[0], radix), min);
>         max = Math.max(Long.parseLong(stats[1], radix), max);
>         sum += Long.parseLong(stats[2], radix);
>         count += Long.parseLong(stats[3], radix);
>       }
>     }
>
>     String ret = Long.toString(min, radix) + "," + Long.toString(max,
> radix) + "," + Long.toString(sum, radix) + "," + Long.toString(count,
> radix);
>     return new Value(ret.getBytes());
>   }
>
>
>
>