Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Minor versions for HFile and HFileBlock


+
ramkrishna vasudevan 2013-06-20, 10:16
+
Ted Yu 2013-06-20, 11:40
+
ramkrishna vasudevan 2013-06-20, 14:25
+
Andrew Purtell 2013-06-20, 16:15
Copy link to this message
-
Re: Minor versions for HFile and HFileBlock
Thanks for the comments.  So i hope the approach that am following is
correct.

Regards
Ram
On Thu, Jun 20, 2013 at 9:45 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> +1 to consolidating these constants into one place (HFile.java?).
>
>
>
>
> On Thu, Jun 20, 2013 at 7:25 AM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> > The FixedFileTrailer uses MAX_MINOR_VERSION.
>  MINOR_VERSION_WITH_FAKED_KEY
> > is being checked to use the INDEX_MAGIC but not sure where it is written
> to
> > the trailer.
> > Some of the Minor versions are spread across the code i felt. Will check
> on
> > that too.
> >
> > Regards
> > Ram
> >
> >
> > On Thu, Jun 20, 2013 at 5:10 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > Take a look at:
> > >
> > >
> >
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> > >
> > >    static final int MAX_MINOR_VERSION = 3;
> > >
> > >    /** Minor versions starting with this number have faked index key */
> > >    static final int MINOR_VERSION_WITH_FAKED_KEY = 3;
> > >
> > > I think the new constant should be added in HFileReaderV2.java so that
> it
> > > is easier to observe all the changes to minor version in one place.
> > >
> > > Cheers
> > >
> > > On Thu, Jun 20, 2013 at 3:16 AM, ramkrishna vasudevan <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > Minor versions are basically for the HFileBlock format.  So any
> change
> > > that
> > > > we try to do in the HFileBlock format we need to increase the minor
> > > > version.
> > > >
> > > > The FixedFileTrailer has the minor version 3 written in it.
> > > > static final int MAX_MINOR_VERSION = 3.
> > > > This forms the min version that is written to the FixedFileTrailer of
> > the
> > > > HFile.
> > > >
> > > > Inside the HFileBlock constructor
> > > > {code}
> > > > if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
> > > >       this.checksumType = b.get();
> > > >       this.bytesPerChecksum = b.getInt();
> > > >       this.onDiskDataSizeWithHeader = b.getInt();
> > > >     } else {
> > > >       this.checksumType = ChecksumType.NULL.getCode();
> > > >       this.bytesPerChecksum = 0;
> > > >       this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
> > > >
> > > >  HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
> > > >     }
> > > > {code}
> > > > where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and
> minorVersion
> > >  > > > > MAX_MINOR_VERSION.
> > > >
> > > > So if i need to make some changes to the HFileBlock format adding
> some
> > > > additional bytes to the header, i need to increase the minorversion
> > > number.
> > > > But since the minor version is already 3 i need to introduce another
> > > > constant which is greater than 3
> > > > and then add the new bytes in the header.
> > > > Something like make
> > > > static final int MAX_MINOR_VERSION = 4.
> > > > Introduce a NEW_CONSTANT =4 in the HFileBlock
> > > > if(minorVersion  >= NEW_CONSTANT)
> > > > {
> > > > //Add new bytes.
> > > > }
> > > > where minorVersion = MAX_MINOR_VERSION .
> > > > So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the
> > new
> > > > one that i introduce now goes to 4. What i was thinking is increasing
> > by
> > > 1
> > > > everywhere in the code should help but seeing the code that is not
> the
> > > > case.
> > > > So is this approach valid or am i missing something?  Testing with
> this
> > > > approach am able to make an existing version of Trunk to work with
> new
> > > > changes that i had made.
> > > >
> > > > Regards
> > > > Ram
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>