Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Master aborts on start up - URGENT


Copy link to this message
-
RE: Master aborts on start up - URGENT
Vladimir Rodionov 2013-07-27, 23:25
There is a parallel thread with a similar issue in user group:
http://mail-archives.apache.org/mod_mbox/hbase-user/201307.mbox/%3CEE3F98CB-A4E8-4BFF-8C5F-AC50E164EB0D%40gmail.com%3E

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: Vladimir Rodionov
Sent: Saturday, July 27, 2013 4:21 PM
To: [EMAIL PROTECTED]
Subject: Master aborts on start up - URGENT

This may be related to :

https://issues.apache.org/jira/browse/HBASE-8912
It has started when I tried to install and run YCSB. I have created 'usertable' and then tried to modify it couple times (added COMPRESSION),
HBase (0.94.6) stopped working (Master could not finish initialization)

I stopped the cluster and physically removed /hbase/usertable directory as well as all ZK local stores. Restarted. No success.

I manually ran OfflineMetaRepair. Restarted. No success. This is  FATAL error in Master's log file.

For some reason, OfflineMetaRepair did not fix missing 'usertable'.

Please, advise. This is a development cluster with a large volume of data.

2013-07-27 23:08:56,504 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: The znode of region TMO_NOV_INDEX-UPLOADS,38,1360181215845.2553b53773e3cb9030c3248768a3b0ca. has been deleted.
2013-07-27 23:08:56,504 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region TMO_NOV_INDEX-UPLOADS,38,1360181215845.2553b53773e3cb9030c3248768a3b0ca. that was online on sjc1-eng-perf-g1-grid06.carrieriq.com,60020,1374966494222
2013-07-27 23:08:56,504 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : usertable,,1374962208806.249881162b6ad6d084b30507283f98b8. state=PENDING_OPEN, ts=1374966536502, server=sjc1-eng-perf-g1-grid14.carrieriq.com,60020,1374966494232 .. Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state : usertable,,1374962208806.249881162b6ad6d084b30507283f98b8. state=PENDING_OPEN, ts=1374966536502, server=sjc1-eng-perf-g1-grid14.carrieriq.com,60020,1374966494232 .. Cannot transit it to OFFLINE.
        at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1820)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1659)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
        at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-07-27 23:08:56,504 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: The znode of region TMO_NOV_INDEX-UPLOADS,46,1360181215846.6f2a2eb3924ba5cb6ed22f966e6356e8. has been deleted.
2013-07-27 23:08:56,505 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: stack (JIRA) [[EMAIL PROTECTED]]
Sent: Saturday, July 27, 2013 3:21 PM
To: [EMAIL PROTECTED]
Subject: [jira] [Created] (HBASE-9063) TestAssignmentManagerOnCluster.testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState fails

stack created HBASE-9063:
----------------------------

             Summary: TestAssignmentManagerOnCluster.testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState fails
                 Key: HBASE-9063
                 URL: https://issues.apache.org/jira/browse/HBASE-9063
             Project: HBase
          Issue Type: Bug
          Components: test
            Reporter: stack
            Assignee: Jimmy Xiang
https://builds.apache.org/job/hbase-0.95-on-hadoop2/200/testReport/org.apache.hadoop.hbase.master/TestAssignmentManagerOnCluster/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/

{code}java.lang.NullPointerException
        at org.apache.hadoop.hbase.master.AssignmentManager.regionOnline(AssignmentManager.java:1314)
        at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState(TestAssignmentManagerOnCluster.java:482)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}

Hope you don't mind my assigning it to you Jimmy.  Thought you might be interested.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Confidentiality Notice:  The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is no