All,
I'm interested in writing a custom iterator, and I've been looking for documentation on how to do so.  Thus far, I've not been able to find anything beyond the java docs in SortedKeyValueIterator and a few other sub-classes.  A few of the examples use Iterators, but provide no real info on how to properly implement one.  Is there anywhere to find general guidance on the iterator stack?

(If you're interested)
Specifically, for those that are curious, I'm trying to implement something similar to the wikisearch example, but with some key differences.  In my case, I've got a file with various attributes that being indexed.  So for each file there are 5 attributes, and each attribute has a fixed number of possible values.  For example (totally made up):
personID, gender, hair color, country, race, personRecord

Row:binID; ColFam:Attribute_AttributeValue; ColQ:PersonID; Val:blank
AND
Row:binID; ColFam:"D"; ColQ:personID; value:personRecord

A typical query would be:
Give me the personRecord for all people with:
Gender: male &
Hair color: blond or brown &
Country: USA or England or china or korea &
Race: white or oriental

The existing Iterators used in the wikisearch example are unable to handle the "or" clauses in each attribute.
The OrIterator doesn't appear to handle the possibility more than one row per tablet

Thanks,
Tejay Cardon

What makes you say that the OrIterator cannot handle more than one row
per tablet? Can you provide details?

AFAIK, the OrIterator should work correctly in all cases (e.g.
regardless of row distribution in a tablet). Any issues in the code that
prevent it from doing so would be a bug that should be fixed.

Also, the wikisearch example supports indexing over multiple attributes
(and I believe indexes document metadata in addition to the tokenized
document). Is there something unclear that could be better documented?

On 8/22/12 4:41 PM, Cardon, Tejay E wrote:
>
> All,
>
> I'm interested in writing a custom iterator, and I've been looking for
> documentation on how to do so.  Thus far, I've not been able to find
> anything beyond the java docs in SortedKeyValueIterator and a few
> other sub-classes.  A few of the examples use Iterators, but provide
> no real info on how to properly implement one.  Is there anywhere to
> find general guidance on the iterator stack?
>
> (If you're interested)
>
> Specifically, for those that are curious, I'm trying to implement
> something similar to the wikisearch example, but with some key
> differences.  In my case, I've got a file with various attributes that
> being indexed.  So for each file there are 5 attributes, and each
> attribute has a fixed number of possible values.  For example (totally
> made up):
>
> personID, gender, hair color, country, race, personRecord
>
> Row:binID; ColFam:Attribute_AttributeValue; ColQ:PersonID; Val:blank
>
> AND
> Row:binID; ColFam:"D"; ColQ:personID; value:personRecord
>
> A typical query would be:
>
> Give me the personRecord for all people with:
>
> Gender: male &
>
> Hair color: blond or brown &
>
> Country: USA or England or china or korea &
>
> Race: white or oriental
>
> The existing Iterators used in the wikisearch example are unable to
> handle the "or" clauses in each attribute.
>
> The OrIterator doesn't appear to handle the possibility more than one
> row per tablet
>
> Thanks,
>
> Tejay Cardon
>

An or clause should be able to handle an enumeration of values, as that's
supported in a JEXL expression. It would not, however, surprise me if those
iterators could not handle multiple rows in a tablet. If you can reproduce
that, please file a ticket. There will be a large update occurring to the
Wiki example in the near future.

Do you have any specific questions about how you should structure your
iterator or the contract? Making a tutorial has been on my to do list, but
we all know how to do lists end up...

The big things to remember are:

1) The call order: Your iterator will be created via the default
constructor, init() will be called, then seek(). After seek() is called,
your iterator should have a top if there is data available. A client then
can call hasTop(), getTopKey() and getTopValue() to check and retrieve data
(similar to hasNext() and next()) and then next to advance the pointer.

2) Your iterator can be destroyed during a scan and then reconstructed,
being passed in the last key returned to the client as the start of the
range.

3) You can have multiple sources feed into a single iterator in a tree like
fashion by clone()'ing the source passed in to init.

On Wed, Aug 22, 2012 at 1:41 PM, Cardon, Tejay E <[EMAIL PROTECTED]>wrote:

>  All,****
>
> I’m interested in writing a custom iterator, and I’ve been looking for
> documentation on how to do so.  Thus far, I’ve not been able to find
> anything beyond the java docs in SortedKeyValueIterator and a few other
> sub-classes.  A few of the examples use Iterators, but provide no real info
> on how to properly implement one.  Is there anywhere to find general
> guidance on the iterator stack?****
>
> ** **
>
> (If you’re interested)****
>
> Specifically, for those that are curious, I’m trying to implement
> something similar to the wikisearch example, but with some key
> differences.  In my case, I’ve got a file with various attributes that
> being indexed.  So for each file there are 5 attributes, and each attribute
> has a fixed number of possible values.  For example (totally made up):****
>
> personID, gender, hair color, country, race, personRecord****
>
> ** **
>
> Row:binID; ColFam:Attribute_AttributeValue; ColQ:PersonID; Val:blank****
>
> AND
> Row:binID; ColFam:”D”; ColQ:personID; value:personRecord****
>
> ** **
>
> A typical query would be:****
>
> Give me the personRecord for all people with:****
>
> Gender: male &****
>
> Hair color: blond or brown &****
>
> Country: USA or England or china or korea &****
>
> Race: white or oriental****
>
> ** **
>
> The existing Iterators used in the wikisearch example are unable to handle
> the “or” clauses in each attribute.****
>
> The OrIterator doesn’t appear to handle the possibility more than one row
> per tablet****
>
> ** **
>
> Thanks,****
>
> Tejay Cardon****
>

Here's a quick write up

    http://www.accumulo.net/node/1 <http://accumulo.net/node/1>

On Wed, Aug 22, 2012 at 8:03 PM, Josh Elser <[EMAIL PROTECTED]> wrote:

> Err, double (triple) reply:
>
> No, you are incorrect. The wikisearch example can handle any arbitrary
> boolean expression containing NOT, AND, and OR. As always, I'll preface it
> the same as Bill did: it *should* be able to handle them :).
>
> I know that cleaning-up/reworking the Wikisearch code is in the works. I'm
> just not positive about the timeframe.
>
> As far as examples, I'd push you to the write-up Eric did after
> benchmarking the wikisearch example: http://accumulo.apache.org/**
> example/wikisearch.html<http://accumulo.apache.org/example/wikisearch.html>
>
> He has some example queries that give the basic idea behind what's
> supported (minus the NOTs)
>
> On 08/22/2012 05:27 PM, Cardon, Tejay E wrote:
>
>>
>> Josh,
>>
>> Thanks for getting back to me so quickly. I explained in my lengthy reply
>> to William that the comment on OrIterator.TermSource.**compareTo
>> indicates that implementations with more than one row per tablet need to
>> compare row key first (and that is not being done in this code). It may be
>> that it’s not an issue and I’m simply misunderstanding something. As for
>> the wikisearch example, as I understood it, it could only handle searches
>> for “anded” terms. If that’s not the case, then an example of an or search
>> would be helpful. In any case, I’d love a deeper dive on the wikisearch
>> somewhere. I get the source code and a high level explanation of what’s
>> happening, but I’d love a tutorial or something that walks through the
>> classes and explains how each one contributes to the functionality. Don’t
>> consider that a request (that would be a lot more to ask then I’m willing
>> to ask), but I would certainly find it useful if it does exist.
>>
>> Thanks,
>>
>> Tejay
>>
>> *From:*Josh Elser [mailto:[EMAIL PROTECTED]]
>> *Sent:* Wednesday, August 22, 2012 2:53 PM
>> *To:* [EMAIL PROTECTED]
>> *Subject:* EXTERNAL: Re: Custom Iterators
>>
>>
>> What makes you say that the OrIterator cannot handle more than one row
>> per tablet? Can you provide details?
>>
>> AFAIK, the OrIterator should work correctly in all cases (e.g. regardless
>> of row distribution in a tablet). Any issues in the code that prevent it
>> from doing so would be a bug that should be fixed.
>>
>> Also, the wikisearch example supports indexing over multiple attributes
>> (and I believe indexes document metadata in addition to the tokenized
>> document). Is there something unclear that could be better documented?
>>
>> On 8/22/12 4:41 PM, Cardon, Tejay E wrote:
>>
>>     All,
>>
>>     I’m interested in writing a custom iterator, and I’ve been looking
>>     for documentation on how to do so. Thus far, I’ve not been able to
>>     find anything beyond the java docs in SortedKeyValueIterator and a
>>     few other sub-classes. A few of the examples use Iterators, but
>>     provide no real info on how to properly implement one. Is there
>>     anywhere to find general guidance on the iterator stack?
>>
>>     (If you’re interested)
>>
>>     Specifically, for those that are curious, I’m trying to implement
>>     something similar to the wikisearch example, but with some key
>>     differences. In my case, I’ve got a file with various attributes
>>     that being indexed. So for each file there are 5 attributes, and
>>     each attribute has a fixed number of possible values. For example
>>     (totally made up):
>>
>>     personID, gender, hair color, country, race, personRecord
>>
>>     Row:binID; ColFam:Attribute_**AttributeValue; ColQ:PersonID;
>> Val:blank
>>
>>     AND
>>     Row:binID; ColFam:”D”; ColQ:personID; value:personRecord
>>
>>     A typical query would be:
>>
>>     Give me the personRecord for all people with:
>>
>>     Gender: male &
>>
>>     Hair color: blond or brown &
>>
>>     Country: USA or England or china or korea &

Err, double (triple) reply:

No, you are incorrect. The wikisearch example can handle any arbitrary
boolean expression containing NOT, AND, and OR. As always, I'll preface
it the same as Bill did: it *should* be able to handle them :).

I know that cleaning-up/reworking the Wikisearch code is in the works.
I'm just not positive about the timeframe.

As far as examples, I'd push you to the write-up Eric did after
benchmarking the wikisearch example:
http://accumulo.apache.org/example/wikisearch.html

He has some example queries that give the basic idea behind what's
supported (minus the NOTs)

On 08/22/2012 05:27 PM, Cardon, Tejay E wrote:
>
> Josh,
>
> Thanks for getting back to me so quickly. I explained in my lengthy
> reply to William that the comment on OrIterator.TermSource.compareTo
> indicates that implementations with more than one row per tablet need
> to compare row key first (and that is not being done in this code). It
> may be that it�s not an issue and I�m simply misunderstanding
> something. As for the wikisearch example, as I understood it, it could
> only handle searches for �anded� terms. If that�s not the case, then
> an example of an or search would be helpful. In any case, I�d love a
> deeper dive on the wikisearch somewhere. I get the source code and a
> high level explanation of what�s happening, but I�d love a tutorial or
> something that walks through the classes and explains how each one
> contributes to the functionality. Don�t consider that a request (that
> would be a lot more to ask then I�m willing to ask), but I would
> certainly find it useful if it does exist.
>
> Thanks,
>
> Tejay
>
> *From:*Josh Elser [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, August 22, 2012 2:53 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* EXTERNAL: Re: Custom Iterators
>
> What makes you say that the OrIterator cannot handle more than one row
> per tablet? Can you provide details?
>
> AFAIK, the OrIterator should work correctly in all cases (e.g.
> regardless of row distribution in a tablet). Any issues in the code
> that prevent it from doing so would be a bug that should be fixed.
>
> Also, the wikisearch example supports indexing over multiple
> attributes (and I believe indexes document metadata in addition to the
> tokenized document). Is there something unclear that could be better
> documented?
>
> On 8/22/12 4:41 PM, Cardon, Tejay E wrote:
>
>     All,
>
>     I�m interested in writing a custom iterator, and I�ve been looking
>     for documentation on how to do so. Thus far, I�ve not been able to
>     find anything beyond the java docs in SortedKeyValueIterator and a
>     few other sub-classes. A few of the examples use Iterators, but
>     provide no real info on how to properly implement one. Is there
>     anywhere to find general guidance on the iterator stack?
>
>     (If you�re interested)
>
>     Specifically, for those that are curious, I�m trying to implement
>     something similar to the wikisearch example, but with some key
>     differences. In my case, I�ve got a file with various attributes
>     that being indexed. So for each file there are 5 attributes, and
>     each attribute has a fixed number of possible values. For example
>     (totally made up):
>
>     personID, gender, hair color, country, race, personRecord
>
>     Row:binID; ColFam:Attribute_AttributeValue; ColQ:PersonID; Val:blank
>
>     AND
>     Row:binID; ColFam:�D�; ColQ:personID; value:personRecord
>
>     A typical query would be:
>
>     Give me the personRecord for all people with:
>
>     Gender: male &
>
>     Hair color: blond or brown &
>
>     Country: USA or England or china or korea &
>
>     Race: white or oriental
>
>     The existing Iterators used in the wikisearch example are unable
>     to handle the �or� clauses in each attribute.
>
>     The OrIterator doesn�t appear to handle the possibility more than
>     one row per tablet
>