Hi,

While merging many regions, I'm getting this error for some of them:

12/12/04 17:45:16 FATAL util.Merge: Merge failed
java.io.IOException: Files have same sequenceid: 75533866
at org.apache.hadoop.hbase.regionserver.HRegion.merge(HRegion.java:4080)
at org.apache.hadoop.hbase.util.Merge.merge(Merge.java:291)
at org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:242)
at org.apache.hadoop.hbase.util.Merge.run(Merge.java:111)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.hbase.util.Merge.main(Merge.java:387)

Any idea why?

Thanks,

JM

Sorry for replying so quickly to myself.

So, here is what I did.

I had a table with only "few" lines". about 20 000.

Table was split over 16 regions.

I merged all the regions into one, then asked HBase via the HTML
interface to split it until I got more than 64 regions.

Then I tried to re-merged them all together again into a single one.

Now, bin/hbase hbck is giving me 65 inconsistencies detected. for this table.

All the inconsistencies are related to the table I played with.

I don't know at what stage the issue happend, so it'S a bit difficult
to reproduce, but seems something went wrong in the process.

JM

2012/12/4, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> Hi,
>
> While merging many regions, I'm getting this error for some of them:
>
> 12/12/04 17:45:16 FATAL util.Merge: Merge failed
> java.io.IOException: Files have same sequenceid: 75533866
> at org.apache.hadoop.hbase.regionserver.HRegion.merge(HRegion.java:4080)
> at org.apache.hadoop.hbase.util.Merge.merge(Merge.java:291)
> at org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:242)
> at org.apache.hadoop.hbase.util.Merge.run(Merge.java:111)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.hbase.util.Merge.main(Merge.java:387)
>
> Any idea why?
>
> Thanks,
>
> JM
>

Regards, Jean-Marc
On 12/04/2012 05:54 PM, Jean-Marc Spaggiari wrote:
> Sorry for replying so quickly to myself.
>
> So, here is what I did.
>
> I had a table with only "few" lines". about 20 000.
>
> Table was split over 16 regions.
>
> I merged all the regions into one, then asked HBase via the HTML
> interface to split it until I got more than 64 regions.
>
> Then I tried to re-merged them all together again into a single one.
>
> Now, bin/hbase hbck is giving me 65 inconsistencies detected. for this table.
>
> All the inconsistencies are related to the table I played with.
>
> I don't know at what stage the issue happend, so it'S a bit difficult
> to reproduce, but seems something went wrong in the process.
>
> JM
>
> 2012/12/4, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
>> Hi,
>>
>> While merging many regions, I'm getting this error for some of them:
>>
>> 12/12/04 17:45:16 FATAL util.Merge: Merge failed
>> java.io.IOException: Files have same sequenceid: 75533866
>> at org.apache.hadoop.hbase.regionserver.HRegion.merge(HRegion.java:4080)
>> at org.apache.hadoop.hbase.util.Merge.merge(Merge.java:291)
>> at org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:242)
>> at org.apache.hadoop.hbase.util.Merge.run(Merge.java:111)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> at org.apache.hadoop.hbase.util.Merge.main(Merge.java:387)
It seems there's an error with Merge process, because it' seems which is
repeating the ID for
files when you repeat the process of Merging regions.
Have you looked in HBASE's JIRAs about this problem?

>>
>> Any idea why?
>>
>> Thanks,
>>
>> JM
>>
>

--

Marcos Luis Ortíz Valmaseda
about.me/marcosortiz <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

One last question, Jean-Marc.
Exactly, What version of HBase are you using?
What version of Hadoop are you using?

On 12/04/2012 09:02 PM, Marcos Ortiz wrote:
> Regards, Jean-Marc
> On 12/04/2012 05:54 PM, Jean-Marc Spaggiari wrote:
>> Sorry for replying so quickly to myself.
>>
>> So, here is what I did.
>>
>> I had a table with only "few" lines". about 20 000.
>>
>> Table was split over 16 regions.
>>
>> I merged all the regions into one, then asked HBase via the HTML
>> interface to split it until I got more than 64 regions.
>>
>> Then I tried to re-merged them all together again into a single one.
>>
>> Now, bin/hbase hbck is giving me 65 inconsistencies detected. for
>> this table.
>>
>> All the inconsistencies are related to the table I played with.
>>
>> I don't know at what stage the issue happend, so it'S a bit difficult
>> to reproduce, but seems something went wrong in the process.
>>
>> JM
>>
>> 2012/12/4, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
>>> Hi,
>>>
>>> While merging many regions, I'm getting this error for some of them:
>>>
>>> 12/12/04 17:45:16 FATAL util.Merge: Merge failed
>>> java.io.IOException: Files have same sequenceid: 75533866
>>>     at
>>> org.apache.hadoop.hbase.regionserver.HRegion.merge(HRegion.java:4080)
>>>     at org.apache.hadoop.hbase.util.Merge.merge(Merge.java:291)
>>>     at
>>> org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:242)
>>>     at org.apache.hadoop.hbase.util.Merge.run(Merge.java:111)
>>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>     at org.apache.hadoop.hbase.util.Merge.main(Merge.java:387)
> It seems there's an error with Merge process, because it' seems which
> is repeating the ID for
> files when you repeat the process of Merging regions.
> Have you looked in HBASE's JIRAs about this problem?
>
>>>
>>> Any idea why?
>>>
>>> Thanks,
>>>
>>> JM
>>>
>>
>

--

Marcos Luis Ortíz Valmaseda
about.me/marcosortiz <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

Hi everyone,

Sorry, I did not figured I was only replying to Marcos ;)

So here are more details about this issue.

I'm using HBase 0.94.3 and Hadoop 1.0.3.

The merge seems to be failing when there is to many merges done.

I just gave it another try... Each time I'm doing a major_compact
before trying the merges. And hbck.

Build the table with 4 regions and 1000 rows. Keys are 8 bytes long
and value is 512 bytes. Everything went well, and hbck is not
reporting additionnal errors.

Another try with 16 regions and 10000 rows worked well too.

Another try with 54 regions and 10000 rows but this one is not working
fine. I'm getting some errors and the hbck is giving that:
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
testtable in hdfs dir
hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94!
It may be an invalid format or version file.  Treating as an orphaned
regiondir.
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
testtable in hdfs dir
hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382!
It may be an invalid format or version file.  Treating as an orphaned
regiondir.
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
testtable in hdfs dir
hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102!
It may be an invalid format or version file.  Treating as an orphaned
regiondir.
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
testtable in hdfs dir
hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd!
It may be an invalid format or version file.  Treating as an orphaned
regiondir.

ERROR: Region { meta => null, hdfs =>
hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94,
deployed =>  } on HDFS, but not listed in META or deployed on any
region server
ERROR: Region { meta => null, hdfs =>
hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382,
deployed =>  } on HDFS, but not listed in META or deployed on any
region server
ERROR: Region { meta => null, hdfs =>
hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102,
deployed =>  } on HDFS, but not listed in META or deployed on any
region server
ERROR: Region { meta => null, hdfs =>
hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd,
deployed =>  } on HDFS, but not listed in META or deployed on any
region server

And below is the log from the merge application. After that it's
listing all the regions in the server (a lot).

I'm not sure if it's the first occurance of the issue or not.

Should I open a JIRA for that? It's difficult to reproduce because
it's not a fixed pattern, but I still can get is failing easily.

I will activate the DEBUG logs on the HRegion class, and give it again
and again another try ;)

Thanks,

JM

Merging testtable,?\xEC\x1B\x90^\xDB\xC9\xA5,1354727393540.3dca6fdbf95546ac71f47403a047fd10.
with testtable,?\xEC\xA0_\xCE+\xB7),1354727402292.a5a42e51f3414fc45d77c1759378b58e.
12/12/05 12:18:56 INFO util.Merge: Verifying that file system is available...
12/12/05 12:18:56 INFO util.Merge: Verifying that HBase is not running...
12/12/05 12:18:56 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=latitude:2181,cube:2181,node3:2181
sessionTimeout=180000 watcher=hconnection
12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Opening socket connection
to server /192.168.23.1:2181
12/12/05 12:18:56 INFO zookeeper.RecoverableZooKeeper: The identifier
of this process is 13131@node3
12/12/05 12:18:56 INFO client.ZooKeeperSaslClient: Client will not
SASL-authenticate because the default JAAS configuration section
'Client' could not be found. If you are not using SASL, you may ignore
this. On the other hand, if you expected SASL to work, please fix your
JAAS configuration.
12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Socket connection
established to cube/192.168.23.1:2181, initiating session
12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Session establishment
complete on server cube/192.168.23.1:2181, sessionid 0x13b6b25f2d900b6, negotiated timeout = 40000
12/12/05 12:18:56 INFO
client.HConnectionManager$HConnectionImplementation: ZooKeeper
available but no active master location found
12/12/05 12:18:56 INFO
client.HConnectionManager$HConnectionImplementation: getMaster attempt
0 of 1 failed; no more retrying.
org.apache.hadoop.hbase.MasterNotRunningException
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:674)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:110)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1733)
at org.apache.hadoop.hbase.util.Merge.run(Merge.java:94)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at MassMerger.mergeAllRegionsFromTable(MassMerger.java:81)
at MassMerger.main(MassMerger.java:120)
12/12/05 12:18:56 INFO
client.HConnectionManager$HConnectionImplementation: Closed zookeeper
sessionid=0x13b6b25f2d900b6
12/12/05 12:18:56 INFO zookeeper.ZooKeeper: Session: 0x13b6b25f2d900b6 closed
12/12/05 12:18:56 INFO zookeeper.ClientCnxn: EventThread shut down
12/12/05 12:18:56 INFO util.Merge: Merging regions
testtable,?\xEC\x1B\x90^\xDB\xC9\xA5,1354727393540.3dca6fdbf95546ac71f47403a047fd10.
and testtable,?\xEC\xA0_\xCE+\xB7),1354727402292.a5a42e51f3414fc45d77c1759378b58e.
in table testtable
12/12/05 12:18:56 INFO wal.HLog: FileSystem doesn't support getDefaultBlockSize
12/12/05 12:18:56 INFO wal.HLog: HLog configuration: blocksize=64 MB,
rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms
12/12/05 12:18:56 INFO wal.HLog:  for
/user/hbase/.logs_1354727936650/hlog.1354727936704
12/12/05 12:18:56 INFO wal.HLog: Using getNumCurrentReplicas--HDFS-826
12/12/05 12:18:56 INFO regionserver.HRegion: Setting up
tabledescriptor config now ...
12/12/05 12:18:56 INFO regionserver.Store: time to purge deletes set
to 0ms in store null
12/12/05 12:18:56 INFO regionserver.HRegion: Onlined
-ROO

Ok. Seems that I'm facing HBASE-1212....

The only issue is that when the merge is failing, hbck become inconsistent.

JM

2012/12/5, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> Hi everyone,
>
> Sorry, I did not figured I was only replying to Marcos ;)
>
> So here are more details about this issue.
>
> I'm using HBase 0.94.3 and Hadoop 1.0.3.
>
> The merge seems to be failing when there is to many merges done.
>
> I just gave it another try... Each time I'm doing a major_compact
> before trying the merges. And hbck.
>
> Build the table with 4 regions and 1000 rows. Keys are 8 bytes long
> and value is 512 bytes. Everything went well, and hbck is not
> reporting additionnal errors.
>
> Another try with 16 regions and 10000 rows worked well too.
>
> Another try with 54 regions and 10000 rows but this one is not working
> fine. I'm getting some errors and the hbck is giving that:
> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
> testtable in hdfs dir
> hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94!
> It may be an invalid format or version file.  Treating as an orphaned
> regiondir.
> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
> testtable in hdfs dir
> hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382!
> It may be an invalid format or version file.  Treating as an orphaned
> regiondir.
> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
> testtable in hdfs dir
> hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102!
> It may be an invalid format or version file.  Treating as an orphaned
> regiondir.
> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
> testtable in hdfs dir
> hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd!
> It may be an invalid format or version file.  Treating as an orphaned
> regiondir.
>
> ERROR: Region { meta => null, hdfs =>
> hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94,
> deployed =>  } on HDFS, but not listed in META or deployed on any
> region server
> ERROR: Region { meta => null, hdfs =>
> hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382,
> deployed =>  } on HDFS, but not listed in META or deployed on any
> region server
> ERROR: Region { meta => null, hdfs =>
> hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102,
> deployed =>  } on HDFS, but not listed in META or deployed on any
> region server
> ERROR: Region { meta => null, hdfs =>
> hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd,
> deployed =>  } on HDFS, but not listed in META or deployed on any
> region server
>
> And below is the log from the merge application. After that it's
> listing all the regions in the server (a lot).
>
> I'm not sure if it's the first occurance of the issue or not.
>
> Should I open a JIRA for that? It's difficult to reproduce because
> it's not a fixed pattern, but I still can get is failing easily.
>
> I will activate the DEBUG logs on the HRegion class, and give it again
> and again another try ;)
>
> Thanks,
>
> JM
>
> Merging
> testtable,?\xEC\x1B\x90^\xDB\xC9\xA5,1354727393540.3dca6fdbf95546ac71f47403a047fd10.
> with
> testtable,?\xEC\xA0_\xCE+\xB7),1354727402292.a5a42e51f3414fc45d77c1759378b58e.
> 12/12/05 12:18:56 INFO util.Merge: Verifying that file system is
> available...
> 12/12/05 12:18:56 INFO util.Merge: Verifying that HBase is not running...
> 12/12/05 12:18:56 INFO zookeeper.ZooKeeper: Initiating client
> connection, connectString=latitude:2181,cube:2181,node3:2181
> sessionTimeout=180000 watcher=hconnection
> 12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Opening socket connection
> to server /192.168.23.1:2181
> 12/12/05 12:18:56 INFO zookeeper.RecoverableZooKeeper: The identifier
> of this process is 13131@node3
> 12/12/05 12:18:56 INFO client.ZooKeeperSaslClient: Client will not
> SASL-authenticate because the default JAAS configuration section
> 'Client' could not be found. If you are not using SASL, you may ignore

When a region merge is failing because of HBASE-1212, it leaves the
system in inconsistency form. I have created HBASE-7287 to fix that
until HBASE-1212 is resolved.

2012/12/5, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> Ok. Seems that I'm facing HBASE-1212....
>
> The only issue is that when the merge is failing, hbck become inconsistent.
>
> JM
>
> 2012/12/5, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
>> Hi everyone,
>>
>> Sorry, I did not figured I was only replying to Marcos ;)
>>
>> So here are more details about this issue.
>>
>> I'm using HBase 0.94.3 and Hadoop 1.0.3.
>>
>> The merge seems to be failing when there is to many merges done.
>>
>> I just gave it another try... Each time I'm doing a major_compact
>> before trying the merges. And hbck.
>>
>> Build the table with 4 regions and 1000 rows. Keys are 8 bytes long
>> and value is 512 bytes. Everything went well, and hbck is not
>> reporting additionnal errors.
>>
>> Another try with 16 regions and 10000 rows worked well too.
>>
>> Another try with 54 regions and 10000 rows but this one is not working
>> fine. I'm getting some errors and the hbck is giving that:
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>>
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>>
>> And below is the log from the merge application. After that it's
>> listing all the regions in the server (a lot).
>>
>> I'm not sure if it's the first occurance of the issue or not.
>>
>> Should I open a JIRA for that? It's difficult to reproduce because
>> it's not a fixed pattern, but I still can get is failing easily.
>>
>> I will activate the DEBUG logs on the HRegion class, and give it again
>> and again another try ;)
>>
>> Thanks,
>>
>> JM
>>
>> Merging
>> testtable,?\xEC\x1B\x90^\xDB\xC9\xA5,1354727393540.3dca6fdbf95546ac71f47403a047fd10.
>> with
>> testtable,?\xEC\xA0_\xCE+\xB7),1354727402292.a5a42e51f3414fc45d77c1759378b58e.
>> 12/12/05 12:18:56 INFO util.Merge: Verifying that file system is
>> available...
>> 12/12/05 12:18:56 INFO util.Merge: Verifying that HBase is not running...
>> 12/12/05 12:18:56 INFO zookeeper.ZooKeeper: Initiating client
>> connection, connectString=latitude:2181,cube:2181,node3:2181
>> sessionTimeout=180000 watcher=hconnection
>> 12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Opening socket connection