Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Issue after Namenode Formating

Copy link to this message
RE: HBase Issue after Namenode Formating
Pankaj Misra 2012-10-17, 22:18
Hi Lars,

Thank you so much Lars for your perfect solution, it worked, you have been a life saver here, worked like a charm. Thanks again Lars.

Pankaj Misra
From: lars hofhansl [[EMAIL PROTECTED]]
Sent: Wednesday, October 17, 2012 11:43 PM
Subject: Re: HBase Issue after Namenode Formating

Can you try clearing all the state in ZK as well? (stop ZK, wipe the ZK data directory, restart ZK)

----- Original Message -----
From: Pankaj Misra <[EMAIL PROTECTED]>
Sent: Wednesday, October 17, 2012 11:09 AM
Subject: RE: HBase Issue after Namenode Formating

Hi All,

I did some more fact finding around this issue, and it seems to me that the Master is not getting initialized and hence I am not able to create table. I tried connecting via the Java client as well, which gave me the following exception. I am not sure what is causing Master to not initialize at all. I am not able to figure out, where am I going wrong, and would request help. Thanks.

java.lang.RuntimeException: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
        at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:56)
        at com.benchmark.uds.hbase.RequestGenerator.main(RequestGenerator.java:72)
Caused by: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:985)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
        at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200)
        at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226)
        at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705)
        at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183)
        at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:233)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:247)
        at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:44)

Also, the error as seen in zookeeper log for the above session is
2012-10-17 23:32:04,274 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client / which had sessionid 0x13a6fd74b50000c
2012-10-17 23:32:05,105 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x13a6fd74b500009, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
        at java.lang.Thread.run(Thread.java:662)
2012-10-17 23:32:05,105 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client / which had sessionid 0x13a6fd74b500009

Thanks and Regards
Pankaj Misra
From: Pankaj Misra
Sent: Wednesday, October 17, 2012 10:30 PM
Subject: RE: HBase Issue after Namenode Formating

Hi All,

I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.

Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below

hbase(main):004:0> create 'testrun','samplecf'

ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
        at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
        at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.

Additionally I see the following error in the HBase Master logs

2012-10-17 21:15:33,043 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry
2012-10-17 21:15:33,045 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for /hbase/backup-masters/localhost,60000,1350488732582 in backup master directory
2012-10-17 21:15:33,083 INFO org.apache.hadoop.hbase.master.ActiveMasterMana