Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Minor versions for HFile and HFileBlock


+
ramkrishna vasudevan 2013-06-20, 10:16
Copy link to this message
-
Re: Minor versions for HFile and HFileBlock
Take a look at:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java

   static final int MAX_MINOR_VERSION = 3;

   /** Minor versions starting with this number have faked index key */
   static final int MINOR_VERSION_WITH_FAKED_KEY = 3;

I think the new constant should be added in HFileReaderV2.java so that it
is easier to observe all the changes to minor version in one place.

Cheers

On Thu, Jun 20, 2013 at 3:16 AM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> Minor versions are basically for the HFileBlock format.  So any change that
> we try to do in the HFileBlock format we need to increase the minor
> version.
>
> The FixedFileTrailer has the minor version 3 written in it.
> static final int MAX_MINOR_VERSION = 3.
> This forms the min version that is written to the FixedFileTrailer of the
> HFile.
>
> Inside the HFileBlock constructor
> {code}
> if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
>       this.checksumType = b.get();
>       this.bytesPerChecksum = b.getInt();
>       this.onDiskDataSizeWithHeader = b.getInt();
>     } else {
>       this.checksumType = ChecksumType.NULL.getCode();
>       this.bytesPerChecksum = 0;
>       this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
>
>  HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
>     }
> {code}
> where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and minorVersion  > MAX_MINOR_VERSION.
>
> So if i need to make some changes to the HFileBlock format adding some
> additional bytes to the header, i need to increase the minorversion number.
> But since the minor version is already 3 i need to introduce another
> constant which is greater than 3
> and then add the new bytes in the header.
> Something like make
> static final int MAX_MINOR_VERSION = 4.
> Introduce a NEW_CONSTANT =4 in the HFileBlock
> if(minorVersion  >= NEW_CONSTANT)
> {
> //Add new bytes.
> }
> where minorVersion = MAX_MINOR_VERSION .
> So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the new
> one that i introduce now goes to 4. What i was thinking is increasing by 1
> everywhere in the code should help but seeing the code that is not the
> case.
> So is this approach valid or am i missing something?  Testing with this
> approach am able to make an existing version of Trunk to work with new
> changes that i had made.
>
> Regards
> Ram
>
+
ramkrishna vasudevan 2013-06-20, 14:25
+
Andrew Purtell 2013-06-20, 16:15
+
ramkrishna vasudevan 2013-06-20, 17:48
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB