Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Minor versions for HFile and HFileBlock


Copy link to this message
-
Re: Minor versions for HFile and HFileBlock
ramkrishna vasudevan 2013-06-20, 14:25
The FixedFileTrailer uses MAX_MINOR_VERSION.  MINOR_VERSION_WITH_FAKED_KEY
is being checked to use the INDEX_MAGIC but not sure where it is written to
the trailer.
Some of the Minor versions are spread across the code i felt. Will check on
that too.

Regards
Ram
On Thu, Jun 20, 2013 at 5:10 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Take a look at:
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
>
>    static final int MAX_MINOR_VERSION = 3;
>
>    /** Minor versions starting with this number have faked index key */
>    static final int MINOR_VERSION_WITH_FAKED_KEY = 3;
>
> I think the new constant should be added in HFileReaderV2.java so that it
> is easier to observe all the changes to minor version in one place.
>
> Cheers
>
> On Thu, Jun 20, 2013 at 3:16 AM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> > Minor versions are basically for the HFileBlock format.  So any change
> that
> > we try to do in the HFileBlock format we need to increase the minor
> > version.
> >
> > The FixedFileTrailer has the minor version 3 written in it.
> > static final int MAX_MINOR_VERSION = 3.
> > This forms the min version that is written to the FixedFileTrailer of the
> > HFile.
> >
> > Inside the HFileBlock constructor
> > {code}
> > if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
> >       this.checksumType = b.get();
> >       this.bytesPerChecksum = b.getInt();
> >       this.onDiskDataSizeWithHeader = b.getInt();
> >     } else {
> >       this.checksumType = ChecksumType.NULL.getCode();
> >       this.bytesPerChecksum = 0;
> >       this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
> >
> >  HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
> >     }
> > {code}
> > where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and minorVersion
>  > > MAX_MINOR_VERSION.
> >
> > So if i need to make some changes to the HFileBlock format adding some
> > additional bytes to the header, i need to increase the minorversion
> number.
> > But since the minor version is already 3 i need to introduce another
> > constant which is greater than 3
> > and then add the new bytes in the header.
> > Something like make
> > static final int MAX_MINOR_VERSION = 4.
> > Introduce a NEW_CONSTANT =4 in the HFileBlock
> > if(minorVersion  >= NEW_CONSTANT)
> > {
> > //Add new bytes.
> > }
> > where minorVersion = MAX_MINOR_VERSION .
> > So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the new
> > one that i introduce now goes to 4. What i was thinking is increasing by
> 1
> > everywhere in the code should help but seeing the code that is not the
> > case.
> > So is this approach valid or am i missing something?  Testing with this
> > approach am able to make an existing version of Trunk to work with new
> > changes that i had made.
> >
> > Regards
> > Ram
> >
>