Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo, mail # user - Accumulo design questions


Copy link to this message
-
Re: Accumulo design questions
Keith Turner 2012-11-06, 19:54
On Tue, Nov 6, 2012 at 2:01 PM, Sukant Hajra <[EMAIL PROTECTED]> wrote:
> I've been trying to understand Accumulo more deeply as we use it more.  To
> supplement the on-line documentation and source, I've been referencing some
> blog articles on HBase (Lars George has some ones), HBase docs, and the
> BigTable paper.
>
> But I'm curious about some of the deviations of Accumulo from BigTable and
> HBase.
>
> The questions I have right now are:
>
>     1. Is the format of an RFile close to HFile version 1, HFile version 2, or
>     at this point is the format really it's own thing?  I found good
>     documentation on the HFile, but I haven't yet found similar documentation
>     on RFiles.  There's the source code, but I haven't dug into that yet.

Looked into this a bit.  Hbase may not have prefix compression (See
HBASE-4676.), which rfile does have a form of.  The R in rfile stands
for relative, each key is encoded relative to the previous one.  If
any field in the key is exactly the same as the previous, then its not
repeated.  So if the row is the same as the row in the previous key,
then its not repeated.   This encoding technique is also utilized when
transferring data over the wire.   Accumulo may do more in the future,
see ACCUMULO-790.

HBASE-4676 also mentions that HFile may not support random access
within a data block.   This exist for RFile in trunk, but is not yet
released.  See ACCUMULO-473.

Seems that HFile v2 may have a multilevel index. See HBASE-4676.
RFile has this.  Prefix compression and multilevel indexes were done
before Accumulo was open sourced, so there is not ticket to ref.

HFile v2 may have some bloom filter improvements that do not exist in
RFile.  I saw these mentioned in something I read about HFile.  I
would need to dig into it some more to see what it is.

Not directly releated but important for end to performance, the C++
map that stores and sorts incoming data is two level.  It has the
following structre.

   map<row, map<column, value>>

This makes inserting a mutation with multiple column updates much much
faster.   This also makes creating RFiles from the map much faster.
Duplicate rows are efficiently transfered to rfile.

>
>     2. I understand that HBase doesn't do well with too many column families.
>     However, creating too many column families in HBase isn't likely anyway
>     because you can't (I believe) create them dynamically.  Accumulo allows you
>     to create column families dynamically.  But I wonder if this can come at a
>     cost.  Is there a benefit to using column families less frequently if
>     possible in Accumulo?  Or is the cost of using column families more or less
>     the same as using column qualifiers.
>
>     3. I guess one way families might be different from qualifiers relates to
>     HBase's recommendation to keep column family names short to avoid needless
>     storage waste.  That should apply to Accumulo as well, right?

As mentioned earlier, relative encoding may help with this some.   If
three consecutive keys have the same column family, then it will only
be written once.

>
>     4. In supporting dynamic column families, was there a design trade-off with
>     respect to the original BigTable or current HBase design?  What might be a
>     benefit of doing it the other way?
>
> Thanks,
> Sukant