Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Problem with HFile lexical comparison


Copy link to this message
-
Re: Problem with HFile lexical comparison
Rohit Kelkar 2013-06-20, 04:08
Perfect. That worked. Thanks.

- R
On Wed, Jun 19, 2013 at 7:23 PM, Jeff Kolesky <[EMAIL PROTECTED]> wrote:

> Last time I wrote directly to an HFile, I instantiated an HFile.Writer
> using this statement:
>
>         HFile.Writer writer = HFile.getWriterFactory(config)
>             .createWriter(fs, hfilePath,
>                     (bytesPerBlock * 1024),
>                     Compression.Algorithm.GZ,
>                     KeyValue.KEY_COMPARATOR);
>
> Perhaps you need the declaration of the comparator in the create statement
> for the writer.
>
> Jeff
>
>
>
> On Wed, Jun 19, 2013 at 5:11 PM, Rohit Kelkar <[EMAIL PROTECTED]>
> wrote:
>
> > Thanks for the replies, I tried the KeyValue.KVComparator but still no
> > luck. So I commented the comparator and played around with the sequence
> of
> > writing the qualifiers to the HFile. (see code here:
> > https://gist.github.com/anonymous/5819254)
> >
> > If I set the variable String[] starr = new String[]{"a", "d", "dt",
> "dth"}
> > then the code breaks while writing the qualifier "dt" to the HFile.
> > If I set the variable String[] starr = new String[]{"a", "dth", "dt",
> "d"}
> > then the code runs successfully.
> > If I set the variable String[] starr = new String[]{"dth", "dt", "d",
> "a"}
> > then the code breaks while writing "a" to the HFile
> >
> > Does this mean that if the qualifiers start with the same character then
> > the longest qualifier should be written first? Else the usual lexical
> order
> > is honoured?
> >
> > The code throws following stack trace
> > Added a key not lexically larger than previous
> >
> >
> key=\x00\x1B10011-2-0000000000000000703\x02sddt\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04,
> >
> >
> lastkey=\x00\x1B10011-2-0000000000000000703\x02sdd\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04
> >  at
> >
> >
> org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:317)
> >  at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:282)
> > at com.mycompany.hbase.process.myprocess.myFunction(MyClass.java:1492)
> >
> > I am using hbase-0.94.2
> >
> > - Rohit Kelkar
> >
> >
> > On Wed, Jun 19, 2013 at 1:15 PM, Jeff Kolesky <[EMAIL PROTECTED]> wrote:
> >
> > > I believe you need to use the KVComparator:
> > >
> > >
> > >
> >
> https://github.com/apache/hbase/blob/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java#L88
> > >
> > > Jeff
> > >
> > >
> > > On Wed, Jun 19, 2013 at 10:32 AM, Rohit Kelkar <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Here is the code - https://gist.github.com/anonymous/5816180
> > > >
> > > > I guess the issue is with my use of the comparator function.
> > > >
> > > > - R
> > > >
> > >
> > >
> > >
> > > --
> > > *Jeff Kolesky*
> > > Chief Software Architect
> > > *Opower*
> > >
> >
>
>
>
> --
> *Jeff Kolesky*
> Chief Software Architect
> *Opower*
>