Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Master server abort


Copy link to this message
-
Re: Master server abort
Enis Söztutar 2013-07-11, 21:30
I've seen a similar stack trace in some test as well, and opened the issue
https://issues.apache.org/jira/browse/HBASE-8912 for tracking this.

This looks like a problem in AssignmentManager that fails to recognize a
valid state transition, but I did not have the time to look into it
further. We'll spend some time to fix this issue, given that this affect
production deployments.

Can you please attach your logs at the issue as well.

Enis
On Thu, Jul 11, 2013 at 11:10 AM, Vladimir Rodionov <[EMAIL PROTECTED]
> wrote:

> This is happening in  one of our small QA cluster.
> HBase 0.94.6.1 (CDH 4.3.0)
>
> 1 master + 5 RS. Zk quorum is 1 (on master node)
>
> We can not start the cluster:
>
> In a log file I find some ERROR's and FATALs . FATAL's come first followed
> by ERRORs (this is important):
>
> FATALs:
>
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> SMALL_GOLDENROD_2012-IDPROFILES,31,1363783108271.77a4640bfaecc907e0ea3535a16c56a8.
> that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> TEST_MM5550_INDEX-UPLOADS,9C,1363689771995.2ee2e6b81ee44ff790abf38275698d45.
> that was online on sjc1-eng-qa03.carrieriq.com,60020,1373485278616
> 2013-07-10 19:42:00,376 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> SMALL_GOLDENROD_2012-IDPROFILES,F8,1363783108280.a0b1b6d003df84ca1404af942bcc9fbc.
> that was online on sjc1-eng-qa02.carrieriq.com,60020,1373485278611
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> TEST_MM5550_INDEX-UPLOADS,E0,1363689771998.36db41b10c86ac537542104c87950709.
> that was online on sjc1-eng-qa06.carrieriq.com,60020,1373485278668
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> SMALL_GOLDENROD_2012-IDPROFILES,FD,1363783108280.95ac4fb83f1bc2753aca0f6a914f6ff2.
> that was online on sjc1-eng-qa02.carrieriq.com,60020,1373485278611
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> TEST_MM5550_INDEX-UPLOADS,E5,1363689771999.65440a41f85b9dd70afd669280491363.
> that was online on sjc1-eng-qa06.carrieriq.com,60020,1373485278668
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> SMALL_GOLDENROD_2012-IDPROFILES,34,1363783108271.979f840723771588cf65910183ecf55c.
> that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> TEST_MM5550_INDEX-UPLOADS,A0,1363689771995.45f193f3dcfcfa76e705b5fa020e4309.
> that was online on sjc1-eng-qa03.carrieriq.com,60020,1373485278616
> 2013-07-10 19:42:00,377 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state :
> packageindex,C0000000,1362756765100.7287ded900b6f6c14f22db5f9ae15d32.
> state=PENDING_OPEN, ts=1373485320376, server=sjc1-eng-qa06.carrieriq.com,60020,1373485278668
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> packageindex,C0000000,1362756765100.7287ded900b6f6c14f22db5f9ae15d32.
> state=PENDING_OPEN, ts=1373485320376, server=sjc1-eng-qa06.carrieriq.com,60020,1373485278668
> .. Cannot transit it to OFFLINE.
>         at
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1820)
>         at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1659)
>         at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
>         at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)