Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # dev >> Combiner Iterator


+
Devin Pinkston 2013-09-18, 13:23
+
Josh Elser 2013-09-18, 14:06
Copy link to this message
-
Re: Combiner Iterator
So, the value iterator is in the abstract class combiner. The super class
of the statscombiner. It automatically passes in the valueiterator to the
reduce method.

in the findTop() method

>         Iterator<Value> viter = new ValueIterator(getSource());
>         topValue = reduce(topKey, viter);
>

this is accomplished by loosening the equality check on the row to exclude
timestamp. meaning rows with similar keys minus the timestamp will be
considered equal.

So, I think, all you need to do is attach your combiner to the table /
column family.
Have you run the statscombiner example on the shell ?

Cheers,
Miguel
On Wed, Sep 18, 2013 at 9:23 AM, Devin Pinkston <[EMAIL PROTECTED]>wrote:

> Hello,
>
> I am trying to work with the example combiner iterator through java code
> instead of the jar or shell.  My question is how do I pass in the
> Iterator<Value> to the reduce method?  Usually I would create a Key
> Value Iterator, but this requires an Iterator just over the Value, and
> then the key to be passed in separately.  The reducer method comes from
> the StatsCombiner class under examples/simple/combiner.  What I have
> right now:
>
>  Iterator<Map.Entry<Key, Value>> iterator = scan.iterator();
>         Iterator<Value> iter;
>
>
>         while (iterator.hasNext()) {
>             Map.Entry<Key, Value> entry = iterator.next();
>             iter = iterator.next().getValue();
>             Key key = entry.getKey();
>             Value value = entry.getValue();
>             reduce(key, iter);
>         }
>
> How would I create the Iterator<Value>?  Every way I have tried has
> led me to dead end or error at runtime.
>
> Thanks!
>
> Here is the reduce method from StatsCombiner.java:
>
>  @Override
>   public Value reduce(Key key, Iterator<Value> iter) {
>
>     long min = Long.MAX_VALUE;
>     long max = Long.MIN_VALUE;
>     long sum = 0;
>     long count = 0;
>
>     while (iter.hasNext()) {
>       String stats[] = iter.next().toString().split(",");
>
>       if (stats.length == 1) {
>         long val = Long.parseLong(stats[0], radix);
>         min = Math.min(val, min);
>         max = Math.max(val, max);
>         sum += val;
>         count += 1;
>       } else {
>         min = Math.min(Long.parseLong(stats[0], radix), min);
>         max = Math.max(Long.parseLong(stats[1], radix), max);
>         sum += Long.parseLong(stats[2], radix);
>         count += Long.parseLong(stats[3], radix);
>       }
>     }
>
>     String ret = Long.toString(min, radix) + "," + Long.toString(max,
> radix) + "," + Long.toString(sum, radix) + "," + Long.toString(count,
> radix);
>     return new Value(ret.getBytes());
>   }
>
>
>
>
+
Devin Pinkston 2013-09-18, 18:35