Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo, mail # user - MinCombiner and MaxCombiner priority issue [SEC=UNCLASSIFIED]


+
Dickson, Matt MR 2013-02-11, 23:47
Copy link to this message
-
Re: MinCombiner and MaxCombiner priority issue [SEC=UNCLASSIFIED]
Adam Fuchs 2013-02-12, 13:23
Hi Matt,

I tried to replicate the behavior you saw and was not able to do so. There
must be some other factors involved. Can you describe what version of
Accumulo you have running and anything else that might be unique about the
instance (other iterators configured on the table, any additional code that
might be in play, etc.)? My test class is listed below, and I ran it with
Accumulo 1.4.2.

Cheers,
Adam
================================= IteratorPriorityTest.java
=============================
import java.util.Collections;
import java.util.Map.Entry;

import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Instance;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.client.ZooKeeperInstance;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.MaxCombiner;
import org.apache.accumulo.core.iterators.user.MinCombiner;
import org.apache.accumulo.core.iterators.user.SummingCombiner;
import org.apache.accumulo.core.security.Authorizations;
import org.apache.hadoop.io.Text;
public class IteratorPriorityTest {

  private static void writeData(int value, BatchWriter writer) throws
Exception {
    Mutation m = new Mutation("harry");
    m.put("count", "1111100000000", "1");
    writer.addMutation(m);
    Mutation m2 = new Mutation("harry");
    m2.put("min", "1111100000000", Integer.toString(value));
    writer.addMutation(m2);
    Mutation m3 = new Mutation("harry");
    m3.put("max", "1111100000000", Integer.toString(value));
    writer.addMutation(m3);
  }

  /**
   * @param args
   */
  public static void main(String[] args) throws Exception {
    Instance inst = new ZooKeeperInstance("instance", "localhost");
    Connector conn = inst.getConnector("root", "password");
    conn.tableOperations().create("test");

    BatchWriter writer = conn.createBatchWriter("test", 1000000, 100, 1);
    writeData(999,writer);
    writeData(12500,writer);
    writeData(1024,writer);
    writeData(2048,writer);
    writer.close();
    Range range = new Range("harry", "harry~");

    //Setup the MIN
    IteratorSetting isTotalMin = new IteratorSetting ( 15, "Min Calc",
MinCombiner.class);
    MinCombiner.setColumns(isTotalMin, Collections.singletonList(new
IteratorSetting.Column("min")));
    MinCombiner.setEncodingType(isTotalMin, MinCombiner.Type.STRING);

    //Setup the MAX
    IteratorSetting isTotalMax = new IteratorSetting ( 14, "Max Calc",
MaxCombiner.class);
    MaxCombiner.setColumns(isTotalMax, Collections.singletonList(new
IteratorSetting.Column("max")));
    MaxCombiner.setEncodingType(isTotalMax, MaxCombiner.Type.STRING);

    //Setup the MIN
    IteratorSetting isTotalCount = new IteratorSetting ( 17, "Count Calc",
SummingCombiner.class);
    SummingCombiner.setColumns(isTotalCount, Collections.singletonList(new
IteratorSetting.Column("count")));
    SummingCombiner.setEncodingType(isTotalCount,
SummingCombiner.Type.STRING);

    Scanner s = conn.createScanner("test", new Authorizations());
    s.addScanIterator(isTotalCount);
    s.addScanIterator(isTotalMin);
    s.addScanIterator(isTotalMax);
    s.setRange(range);
    s.fetchColumnFamily(new Text("count"));
    s.fetchColumnFamily(new Text("min"));
    s.fetchColumnFamily(new Text("max"));
    for (Entry<Key, Value> e : s) {
      System.out.println(e.getKey().getRow() + ", " +
e.getKey().getColumnFamily() + ", " + e.getKey().getColumnQualifier() + ",
VALUE: " + e.getValue());
    }

    conn.tableOperations().delete("test");
  }

}

=============================================================================================

On Mon, Feb 11, 2013 at 6:47 PM, Dickson, Matt MR <
[EMAIL PROTECTED]> wrote:

> **
>
> *UNCLASSIFIED*
>  Hi,