Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - DISCUSS : HFile V3 proposal for tags in 0.96


+
ramkrishna vasudevan 2013-07-18, 17:14
+
Ted Yu 2013-07-18, 17:23
Copy link to this message
-
Re: DISCUSS : HFile V3 proposal for tags in 0.96
Jimmy Xiang 2013-07-18, 17:55
Can you share some more details about it?  A graph/chart/table showing the
specific difference will be helpful.

Thanks,
Jimmy
On Thu, Jul 18, 2013 at 10:23 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> I have been following comments on HBASE-8496.
>
> I think introducing cell tagging through HFile v3 is acceptable.
>
> Looking forward to seeing your implementation.
>
> Cheers
>
> On Thu, Jul 18, 2013 at 10:14 AM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> > For the past couple of months, we have been working through various
> > prototypes for supporting inline storage of tags in cells as persisted on
> > disk. Our goals are to support optional use of tags with minimal changes
> to
> > core code while also avoiding performance impacts to users who do not use
> > tags.
> >
> >  For background, refer to the comments in
> >
> >
> >
> https://issues.apache.org/jira/browse/HBASE-8496?focusedCommentId=13708228&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13708228
> >
> > and
> >
> >
> >
> https://issues.apache.org/jira/browse/HBASE-8496?focusedCommentId=13710653&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13710653
> >
> >  We have iterated on a couple of prototypes that implement tag awareness
> in
> > DataBlockEncoders, later as a new type of Codec for Cells. This point is
> > discussed in the above comments in HBASE-8496.
> >
> > We think that tag awareness in Cell Codecs is the right way, but there
> are
> > some shortcomings with the current interfaces internal to HFile that need
> > to addressed in order to avoid any performance impacts for those who do
> not
> > want to use inline tags, and that may involve a drastic amount of code
> > change.
> >
> >  We can avoid several problems with HFile V2 internals, and backwards
> > compatibility concerns, and allow for working tags support with no
> > performance impact and low risk to all HBase users who do not want tag
> > support, while still allowing for inline tags capabilities in a shipping
> > version of HBase, by introducing this in a new V3 version for HFile.
> >
> >  The new V3 version for HFile differs from earlier versions by supporting
> > inline tag storage.  This version does not change the HFileBlock format
> > whereas it just serializes and deserializes the Tag information that
> would
> > be persisted in the HFile. Having HFile V3 would also help to keep Tags
> > optional such that the existing cases where there are no tags are totally
> > unaffected.  Also we ensure that we keep the changes outside of the V3
> > reader and writer minimal.  Compatibility would not be a problem with
> > future versions when we go with Cell Codecs.  What Codecs used for
> writing
> > the file will be persisted in the HFile header.  Now for files that are
> > either V2 or V3 we will instantiate two default codecs that know to deal
> > with serializations with and without tags.
> >
> >  There have been thoughts on an HFile V3 prior, e.g.:
> >
> >
> >
> https://issues.apache.org/jira/browse/HBASE-8496?focusedCommentId=13710653&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13710653
> >
> >  We have been working on this and will have a clean patch with good
> amount
> > of testing in time for 0.96.
> >
> > Although our focus is on performance-neutral persistence of inline cell
> > tags in 0.96 to enable a couple of security coprocessor users,
> introducing
> > an HFile V3 provides design freedom for some other features and problems
> > too that can be developed through the 0.96 cycle into 0.98.
> >
> > Pls voice your opinion on this so that we can make this clear and may be
> > define the scope of the patch.  Also feel free to comment on HBASE-8496
> on
> > your thoughts and ideas.
> >
> > Regards
> >
> > Ram
> >
>
+
ramkrishna vasudevan 2013-07-19, 04:29
+
Ted Yu 2013-07-19, 04:40
+
ramkrishna vasudevan 2013-07-19, 04:57
+
Ted Yu 2013-07-19, 05:00
+
Stack 2013-07-19, 05:12
+
Jean-Marc Spaggiari 2013-07-19, 11:02
+
ramkrishna vasudevan 2013-07-19, 11:11
+
Jean-Marc Spaggiari 2013-07-19, 11:23
+
Anoop John 2013-07-19, 11:51
+
ramkrishna vasudevan 2013-07-19, 12:00
+
Jean-Marc Spaggiari 2013-07-19, 13:09
+
Ted Yu 2013-07-19, 14:18
+
Anoop John 2013-07-19, 15:12
+
ramkrishna vasudevan 2013-07-19, 15:05
+
Andrew Purtell 2013-07-19, 16:27
+
Ted Yu 2013-07-19, 16:32
+
Anoop John 2013-07-19, 17:13
+
Ted Yu 2013-07-19, 17:35
+
Elliott Clark 2013-07-19, 17:52
+
Andrew Purtell 2013-07-19, 18:01
+
Elliott Clark 2013-07-19, 21:02
+
Andrew Purtell 2013-07-19, 22:34
+
Stack 2013-07-19, 23:31
+
ramkrishna vasudevan 2013-07-20, 02:10
+
Andrew Purtell 2013-07-22, 17:23
+
Jean-Marc Spaggiari 2013-07-23, 22:43
+
Andrew Purtell 2013-07-24, 17:33
+
ramkrishna vasudevan 2013-07-25, 18:09
+
Andrew Purtell 2013-07-19, 16:48
+
Ted Yu 2013-07-24, 17:30
+
Andrew Purtell 2013-07-19, 16:25