Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> java.io.IOEcxeption key k1 followed by a smaller key k2


+
Mohamed Ibrahim 2012-09-16, 12:59
+
Stack 2012-09-16, 22:09
+
lars hofhansl 2012-09-16, 23:33
+
lars hofhansl 2012-09-17, 04:04
Copy link to this message
-
Re: java.io.IOEcxeption key k1 followed by a smaller key k2
Hello Lars / Stack,

Thank you for responding.

The date on the files is March 9th 2012. It's been up since then, I
restarted hbase and hadoop once. I only have a single node that I'm running
my tests on. I'm currently running 0.92.1 on hadoop 1.0 . I hope I'm using
the correct mix.

I'm not using any external tools other than hbase java api, and inspecting
the data using the shell. I ran my program, the following day I found the
stack dump on the console. I checked by scanning the table that had the
exception for the smaller key k1, with limit 2 from the shell and no
exceptions were thrown. I can also see that the following key is larger
than the first one, so nothing is wrong.

I faced the same exception before and it happened more frequently when I
did a lot of Puts and Deletes using Htable.batch . Essentially I was
updating several inverted indexes each in its own table on the data of rows
when they get inserted, and instead of doing single Deletes & Puts on the
indexes I used batch. batch improved the performance but threw this
exception more. I stopped using batch and now doing single Puts & Deletes.
The exception rarely gets thrown, but still sporadically gets thrown.

I will read about the hfile tool, thanks for the pointers. I will also try
figure out a set of steps that would repeat the exception so it is more
helpful. I will also try 0.94.1 with batch and see if it will happen again,
and will let you know and will file a bug if I can repeat it consistently.

Thank you,
Mohamed Ibrahim

On Sun, Sep 16, 2012 at 7:33 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> Hmm... HBASE-6579 gets rid of that check, because we thought it no longer
> necessary.
> No do you remember what you did leading up to this?
> Did you write these HFiles with some other tool? Done some bulk import,
> etc?
>
>
> -- Lars
>
>
>
> ________________________________
>  From: Mohamed Ibrahim <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Sunday, September 16, 2012 5:59 AM
> Subject: java.io.IOEcxeption key k1 followed by a smaller key k2
>
> Hello All,
>
> I am using hbase 0.92.1 on hadoop 1 . I am getting those exceptions, and it
> seems to me that it means that the hbase file is not sorted in order. So
> when the scanner goes through it, it finds a smaller key after its current.
>
> Is that related to https://issues.apache.org/jira/browse/HBASE-6579 ??
>
> It looks like upgrading to 0.94.1 (current stable) won't fix the issue. Any
> recommendations ??
>
> Here is the stack dump:
>         at
>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:266)
>         at
>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:138)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:2945)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:2901)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:2918)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.getLastIncrement(HRegion.java:3660)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.incrementColumnValue(HRegion.java:3844)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.incrementColumnValue(HRegionServer.java:2982)
>         at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
>
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
>         at
> org.apache.hadoop.hbase.client.HTable.incrementColumnValue(HTable.java:870)
+
lars hofhansl 2012-09-17, 04:06
+
Mohamed Ibrahim 2012-09-17, 17:00