Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo, mail # user - Custom Iterators - behavior when switching tablets


Copy link to this message
-
Re: Custom Iterators - behavior when switching tablets
Adam Fuchs 2013-01-23, 15:45
David,

The core challenge here is to be able to continue scans under failure
conditions. There are several places where we tear down the iterator tree
and rebuild it, including when tablet servers die, when we need to free
resources to support concurrency, and a few others. In order to continue a
scan where we left off, we need to be able to point to some place in the
stream of key/value pairs. If we want to be robust against tablet server
failure we can't just store that scan session information on the tablet
server, so we use the last key that was returned to the client for that
information.

When you add an iterator that transforms keys, you change the meaning of
that pointer so that it points into the transformed stream instead of the
underlying stream. There are two requirements in order to do this sanely:
1. Your iterator should not change the row portion of the key, although it
can change any of the remaining parts.
2. Your iterator's seek method should perform the reverse transformation on
the range when it seeks the underlying iterator(s). This will ensure that
you don't skip ranges of underlying keys when the scanner continues the
scan.

That said, I agree that the behavior of the scanner where it ignores keys
that are returned is probably not optimal, and I'm not sure why it does
that, except maybe to prevent some infinite loops.

Adam

On Tue, Jan 22, 2013 at 11:55 AM, Slater, David M.
<[EMAIL PROTECTED]>wrote:

> In designing some of my own custom iterators, I was noticing some
> interesting behavior. Note: my iterator does not return the original key,
> but instead returns a computed value that is not necessarily in
> lexicographic order.****
>
> ** **
>
> So far as I can tell, when the Scanner switches between tablets, it checks
> the key that is returned in the new tablet and compares it (I think it
> compares key.row()) with the last key from the previous tablet. If the new
> key is greater than the previous one, then it proceeds normally. If,
> however, the new key is less than or equal to the previous key, then the
> Scanner does not return the value. It does, however, continue to iterate
> through the tablet, continuing to compare until it finds a key greater than
> the last one. Once it finds one, however, it progresses through the rest of
> that tablet without doing a check. (It implicitly assumes that everything
> in a tablet will be correctly ordered). ****
>
> ** **
>
> Now if I was to return the original key, it would work fine (since it
> would always be in order), but that also limits the functionality of my
> custom iterator. ****
>
> ** **
>
> My primary question is: why would it be designed this way? When switching
> between tablets, are there potential problems that might crop up if this
> check isn’t done?****
>
> ** **
>
> Thanks,
> David****
>