I've been trying to understand Accumulo more deeply as we use it more.  To
supplement the on-line documentation and source, I've been referencing some
blog articles on HBase (Lars George has some ones), HBase docs, and the
BigTable paper.

But I'm curious about some of the deviations of Accumulo from BigTable and
HBase.

The questions I have right now are:

    1. Is the format of an RFile close to HFile version 1, HFile version 2, or
    at this point is the format really it's own thing?  I found good
    documentation on the HFile, but I haven't yet found similar documentation
    on RFiles.  There's the source code, but I haven't dug into that yet.

    2. I understand that HBase doesn't do well with too many column families.
    However, creating too many column families in HBase isn't likely anyway
    because you can't (I believe) create them dynamically.  Accumulo allows you
    to create column families dynamically.  But I wonder if this can come at a
    cost.  Is there a benefit to using column families less frequently if
    possible in Accumulo?  Or is the cost of using column families more or less
    the same as using column qualifiers.

    3. I guess one way families might be different from qualifiers relates to
    HBase's recommendation to keep column family names short to avoid needless
    storage waste.  That should apply to Accumulo as well, right?

    4. In supporting dynamic column families, was there a design trade-off with
    respect to the original BigTable or current HBase design?  What might be a
    benefit of doing it the other way?

Thanks,
Sukant

1. I'm not too familir with the differenet HFile versions, I really can't
draw comparisons. But RFiles were developed independently, but I think at a
high level they share the same idea.

2. Column families do not have the performance penalty they have in HBase
because we do not have a 1:1 ratio of column families to locality groups.
The only real benefit I forsee from minimizing column families is a bit
more compaction due to the relative encoding in RFiles. But for all intents
and purposes, you can think of them as a similar performance metric to that
of column qualifiers, until you start creating locality groups from them.

3. Disk is cheap and the second you start trying to shave a byte here and a
few bits there you're going against the natural design of the architecture.
You want the data to be first usable, which is something you can lose once
you start having ellaborate compaction schemes. Again, the relative
encoding + standard compression done in the RFiles should be more than
enough for making your data tiny enough. Also, I would not be surprised if
the logic behind long column families in HBase is due to the locality group
issue, which as mentioned above, is not an issue for Accumulo.

4. I believe the BigTable paper specified 'a mapping from column families
to a locality group', which is more in line with our configuration.
However, BigTable also specified that all column families are defined in
advance, which is more in tune with Hbase. We feel the dynamic nature of
our system provides enough flexibility to be convenient to use while still
providing mechanisms to harness the power of locality groups. In standard
Accumulo use cases, switching to the other way would probably be a
hindrance because we don't try to minimize column families, which means
more blocks needing to be merged together at scan time, creating a
significant performance hit.

Hope this answers some questions

John
On Tue, Nov 6, 2012 at 2:01 PM, Sukant Hajra <[EMAIL PROTECTED]> wrote:

> I've been trying to understand Accumulo more deeply as we use it more.  To
> supplement the on-line documentation and source, I've been referencing some
> blog articles on HBase (Lars George has some ones), HBase docs, and the
> BigTable paper.
>
> But I'm curious about some of the deviations of Accumulo from BigTable and
> HBase.
>
> The questions I have right now are:
>
>     1. Is the format of an RFile close to HFile version 1, HFile version
> 2, or
>     at this point is the format really it's own thing?  I found good
>     documentation on the HFile, but I haven't yet found similar
> documentation
>     on RFiles.  There's the source code, but I haven't dug into that yet.
>
>     2. I understand that HBase doesn't do well with too many column
> families.
>     However, creating too many column families in HBase isn't likely anyway
>     because you can't (I believe) create them dynamically.  Accumulo
> allows you
>     to create column families dynamically.  But I wonder if this can come
> at a
>     cost.  Is there a benefit to using column families less frequently if
>     possible in Accumulo?  Or is the cost of using column families more or
> less
>     the same as using column qualifiers.
>
>     3. I guess one way families might be different from qualifiers relates
> to
>     HBase's recommendation to keep column family names short to avoid
> needless
>     storage waste.  That should apply to Accumulo as well, right?
>
>     4. In supporting dynamic column families, was there a design trade-off
> with
>     respect to the original BigTable or current HBase design?  What might
> be a
>     benefit of doing it the other way?
>
> Thanks,
> Sukant
>

On Tue, Nov 6, 2012 at 2:01 PM, Sukant Hajra <[EMAIL PROTECTED]> wrote:
> I've been trying to understand Accumulo more deeply as we use it more.  To
> supplement the on-line documentation and source, I've been referencing some
> blog articles on HBase (Lars George has some ones), HBase docs, and the
> BigTable paper.
>
> But I'm curious about some of the deviations of Accumulo from BigTable and
> HBase.
>
> The questions I have right now are:
>
>     1. Is the format of an RFile close to HFile version 1, HFile version 2, or
>     at this point is the format really it's own thing?  I found good
>     documentation on the HFile, but I haven't yet found similar documentation
>     on RFiles.  There's the source code, but I haven't dug into that yet.

Looked into this a bit.  Hbase may not have prefix compression (See
HBASE-4676.), which rfile does have a form of.  The R in rfile stands
for relative, each key is encoded relative to the previous one.  If
any field in the key is exactly the same as the previous, then its not
repeated.  So if the row is the same as the row in the previous key,
then its not repeated.   This encoding technique is also utilized when
transferring data over the wire.   Accumulo may do more in the future,
see ACCUMULO-790.

HBASE-4676 also mentions that HFile may not support random access
within a data block.   This exist for RFile in trunk, but is not yet
released.  See ACCUMULO-473.

Seems that HFile v2 may have a multilevel index. See HBASE-4676.
RFile has this.  Prefix compression and multilevel indexes were done
before Accumulo was open sourced, so there is not ticket to ref.

HFile v2 may have some bloom filter improvements that do not exist in
RFile.  I saw these mentioned in something I read about HFile.  I
would need to dig into it some more to see what it is.

Not directly releated but important for end to performance, the C++
map that stores and sorts incoming data is two level.  It has the
following structre.

   map<row, map<column, value>>

This makes inserting a mutation with multiple column updates much much
faster.   This also makes creating RFiles from the map much faster.
Duplicate rows are efficiently transfered to rfile.

>
>     2. I understand that HBase doesn't do well with too many column families.
>     However, creating too many column families in HBase isn't likely anyway
>     because you can't (I believe) create them dynamically.  Accumulo allows you
>     to create column families dynamically.  But I wonder if this can come at a
>     cost.  Is there a benefit to using column families less frequently if
>     possible in Accumulo?  Or is the cost of using column families more or less
>     the same as using column qualifiers.
>
>     3. I guess one way families might be different from qualifiers relates to
>     HBase's recommendation to keep column family names short to avoid needless
>     storage waste.  That should apply to Accumulo as well, right?

As mentioned earlier, relative encoding may help with this some.   If
three consecutive keys have the same column family, then it will only
be written once.

>
>     4. In supporting dynamic column families, was there a design trade-off with
>     respect to the original BigTable or current HBase design?  What might be a
>     benefit of doing it the other way?
>
> Thanks,
> Sukant

On Tue, Nov 6, 2012 at 11:01 AM, Sukant Hajra <[EMAIL PROTECTED]>wrote:

> I've been trying to understand Accumulo more deeply as we use it more.  To
> supplement the on-line documentation and source, I've been referencing some
> blog articles on HBase (Lars George has some ones), HBase docs, and the
> BigTable paper.
>
> But I'm curious about some of the deviations of Accumulo from BigTable and
> HBase.
>
> The questions I have right now are:
>
>     1. Is the format of an RFile close to HFile version 1, HFile version
> 2, or
>     at this point is the format really it's own thing?  I found good
>     documentation on the HFile, but I haven't yet found similar
> documentation
>     on RFiles.  There's the source code, but I haven't dug into that yet.
>

I think there is a different HFile for each column family, isn't there?  An
RFile stores all columns, all locality groups in a single file, which is
another reason you don't get the same performance penalty for having lots
of column families in Accumulo.
>
>     2. I understand that HBase doesn't do well with too many column
> families.
>     However, creating too many column families in HBase isn't likely anyway
>     because you can't (I believe) create them dynamically.  Accumulo
> allows you
>     to create column families dynamically.  But I wonder if this can come
> at a
>     cost.  Is there a benefit to using column families less frequently if
>     possible in Accumulo?  Or is the cost of using column families more or
> less
>     the same as using column qualifiers.
>
>     3. I guess one way families might be different from qualifiers relates
> to
>     HBase's recommendation to keep column family names short to avoid
> needless
>     storage waste.  That should apply to Accumulo as well, right?
>
>     4. In supporting dynamic column families, was there a design trade-off
> with
>     respect to the original BigTable or current HBase design?  What might
> be a
>     benefit of doing it the other way?
>

The main thing Accumulo had to do differently from BigTable to allow
dynamic creation of column families was to create a default locality
group.  That's the locality group that stores column families that aren't
specified for any other locality group.  I recall Keith saying it was kind
of a pain to implement, but I don't see any obvious negative tradeoffs of
the design.

Billie

> Thanks,
> Sukant
>

On Tue, Nov 6, 2012 at 3:30 PM, Billie Rinaldi <[EMAIL PROTECTED]> wrote:
> On Tue, Nov 6, 2012 at 11:01 AM, Sukant Hajra <[EMAIL PROTECTED]>
> wrote:
>>
>> I've been trying to understand Accumulo more deeply as we use it more.  To
>> supplement the on-line documentation and source, I've been referencing
>> some
>> blog articles on HBase (Lars George has some ones), HBase docs, and the
>> BigTable paper.
>>
>> But I'm curious about some of the deviations of Accumulo from BigTable and
>> HBase.
>>
>> The questions I have right now are:
>>
>>     1. Is the format of an RFile close to HFile version 1, HFile version
>> 2, or
>>     at this point is the format really it's own thing?  I found good
>>     documentation on the HFile, but I haven't yet found similar
>> documentation
>>     on RFiles.  There's the source code, but I haven't dug into that yet.
>
>
> I think there is a different HFile for each column family, isn't there?  An
> RFile stores all columns, all locality groups in a single file, which is
> another reason you don't get the same performance penalty for having lots of
> column families in Accumulo.
>
>>
>>
>>     2. I understand that HBase doesn't do well with too many column
>> families.
>>     However, creating too many column families in HBase isn't likely
>> anyway
>>     because you can't (I believe) create them dynamically.  Accumulo
>> allows you
>>     to create column families dynamically.  But I wonder if this can come
>> at a
>>     cost.  Is there a benefit to using column families less frequently if
>>     possible in Accumulo?  Or is the cost of using column families more or
>> less
>>     the same as using column qualifiers.
>>
>>     3. I guess one way families might be different from qualifiers relates
>> to
>>     HBase's recommendation to keep column family names short to avoid
>> needless
>>     storage waste.  That should apply to Accumulo as well, right?
>>
>>     4. In supporting dynamic column families, was there a design trade-off
>> with
>>     respect to the original BigTable or current HBase design?  What might
>> be a
>>     benefit of doing it the other way?
>
>
> The main thing Accumulo had to do differently from BigTable to allow dynamic
> creation of column families was to create a default locality group.  That's
> the locality group that stores column families that aren't specified for any
> other locality group.  I recall Keith saying it was kind of a pain to
> implement, but I don't see any obvious negative tradeoffs of the design.

It seems like with Big Table you can drop a locality group and all of
the data related to the locality group goes away. Even if Big Table
does not support this, that would be true of the BigTable model.
With default locality groups, if you drop a locality group, then that
data will end up in the default locality group.  This is not a
negative, but a difference.

One other point, Accumulo supports online changes to locality group
configuration.  If you change the locality group config for a table,
then all files created after that point will use the new config.  This
is easy to do because each RFile encapsulates a set of locality groups
as Billie mentioned.  So the locality group config goes with the file,
nothing external is needed by Rfile to make smart decisions when
seeking.

>
> Billie
>
>
>>
>> Thanks,
>> Sukant
>
>

>     4. In supporting dynamic column families, was there a design trade-off
> with
>     respect to the original BigTable or current HBase design?  What might
> be a
>     benefit of doing it the other way?
>
> One trade-off is that pinning locality groups in memory (i.e. making them
ephemeral) would be challenging for Accumulo, while this is something that
Bigtable supports.

Another trade-off is that supporting compactions on only one locality group
at a time would be impossible. Since any one file can hold all locality
groups, it is likely that all data will go through the compaction channels
even if only one locality group is being actively written. The flip side of
this is that Accumulo uses fewer files, putting less load on HDFS.

Adam