Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Minor versions for HFile and HFileBlock


Copy link to this message
-
Re: Minor versions for HFile and HFileBlock
+1 to consolidating these constants into one place (HFile.java?).
On Thu, Jun 20, 2013 at 7:25 AM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> The FixedFileTrailer uses MAX_MINOR_VERSION.  MINOR_VERSION_WITH_FAKED_KEY
> is being checked to use the INDEX_MAGIC but not sure where it is written to
> the trailer.
> Some of the Minor versions are spread across the code i felt. Will check on
> that too.
>
> Regards
> Ram
>
>
> On Thu, Jun 20, 2013 at 5:10 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > Take a look at:
> >
> >
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> >
> >    static final int MAX_MINOR_VERSION = 3;
> >
> >    /** Minor versions starting with this number have faked index key */
> >    static final int MINOR_VERSION_WITH_FAKED_KEY = 3;
> >
> > I think the new constant should be added in HFileReaderV2.java so that it
> > is easier to observe all the changes to minor version in one place.
> >
> > Cheers
> >
> > On Thu, Jun 20, 2013 at 3:16 AM, ramkrishna vasudevan <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Minor versions are basically for the HFileBlock format.  So any change
> > that
> > > we try to do in the HFileBlock format we need to increase the minor
> > > version.
> > >
> > > The FixedFileTrailer has the minor version 3 written in it.
> > > static final int MAX_MINOR_VERSION = 3.
> > > This forms the min version that is written to the FixedFileTrailer of
> the
> > > HFile.
> > >
> > > Inside the HFileBlock constructor
> > > {code}
> > > if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
> > >       this.checksumType = b.get();
> > >       this.bytesPerChecksum = b.getInt();
> > >       this.onDiskDataSizeWithHeader = b.getInt();
> > >     } else {
> > >       this.checksumType = ChecksumType.NULL.getCode();
> > >       this.bytesPerChecksum = 0;
> > >       this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
> > >
> > >  HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
> > >     }
> > > {code}
> > > where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and minorVersion
> >  > > > MAX_MINOR_VERSION.
> > >
> > > So if i need to make some changes to the HFileBlock format adding some
> > > additional bytes to the header, i need to increase the minorversion
> > number.
> > > But since the minor version is already 3 i need to introduce another
> > > constant which is greater than 3
> > > and then add the new bytes in the header.
> > > Something like make
> > > static final int MAX_MINOR_VERSION = 4.
> > > Introduce a NEW_CONSTANT =4 in the HFileBlock
> > > if(minorVersion  >= NEW_CONSTANT)
> > > {
> > > //Add new bytes.
> > > }
> > > where minorVersion = MAX_MINOR_VERSION .
> > > So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the
> new
> > > one that i introduce now goes to 4. What i was thinking is increasing
> by
> > 1
> > > everywhere in the code should help but seeing the code that is not the
> > > case.
> > > So is this approach valid or am i missing something?  Testing with this
> > > approach am able to make an existing version of Trunk to work with new
> > > changes that i had made.
> > >
> > > Regards
> > > Ram
> > >
> >
>

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)