Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Cell Encoders and usage of Cell


Copy link to this message
-
Re: Cell Encoders and usage of Cell
ramkrishna vasudevan 2013-04-23, 06:40
>Until then, you are left w/ some sort of hack on KV.

Yes.
> If all was Cell on the read/write path, I'd imagine
it'd be easy enough getting your tags in the mix.
Am checking on this also.  May take some time till i finalise on the
changes.

Will discuss with Andy offline and get back on some initial version on the
changes that may be needed.

Regards
Ram
On Tue, Apr 23, 2013 at 3:46 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> On HBASE-7544 I added encryption to HFile starting with the encoder and
> decoder contexts and working outward, so I can attest it is possible to add
> features that wrap *around* KV serialization without changing the KV
> serialization itself. On trunk the block encoder stuff has been at least
> partially factored out so it is not so bad. The HFileV2 code in 0.94 is
> more of a hairball.
>
> To pursue an incremental development approach, one option to consider is
> making it possible to stack block encoders. Then we might be able to
> implement tag support as a block encoder without losing the ability to do
> real block encoding. Something else to undo is the inflexible enum that
> specifies block encoder types. We should use a short int signature or
> something instead. Think of treating block encoders like filters. Then
> encoder/decoder stacking could be assembled at runtime from metadata read
> out of hfile.
>
>
> On Mon, Apr 22, 2013 at 12:28 PM, Stack <[EMAIL PROTECTED]> wrote:
>
> > On Thu, Apr 18, 2013 at 8:19 PM, ramkrishna vasudevan <
> > [EMAIL PROTECTED]> wrote:
> >
> > > My questions were mainly because, if i have the current code  and i
> would
> > > want to introduce tags in it, where would i do it?
> > >
> >
> >
> > Good question.  If the read/write path currently does not support
> Cell-only
> > access, you are in a bit of a bind.
> >
> > It would take some effort converting all to pure Cell.  Most would be
> just
> > grunt work but there are a few tricky spots (according to a quick survey
> > done by our LarsH).  If all was Cell on the read/write path, I'd imagine
> > it'd be easy enough getting your tags in the mix.
> >
> > Until then, you are left w/ some sort of hack on KV.
> >
> >
> >
> > > So if i need tags to be introduced should i start changing the HFile
> > > formats also and only then i would be getting the tags to work?
> > > What do you think here?
> > >
> > >
> > hfiles are versioned so we can up the version when we put a Cell API on
> it.
> >  Might be a bit of work though since will have to bring along the
> > compressors and block encoders and currently these are hard-coded to
> expect
> > KV and in particular KVs current serialization.
> >
> > St.Ack
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>