Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Issue after Namenode Formating


Copy link to this message
-
Re: HBase Issue after Namenode Formating
No problem...

We need to fix this in HBase too.There's HBASE-6294, but no patch for it, yet.

-- Lars

________________________________
 From: Pankaj Misra <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>
Sent: Wednesday, October 17, 2012 3:18 PM
Subject: RE: HBase Issue after Namenode Formating
 
Hi Lars,

Thank you so much Lars for your perfect solution, it worked, you have been a life saver here, worked like a charm. Thanks again Lars.

Regards
Pankaj Misra
________________________________________
From: lars hofhansl [[EMAIL PROTECTED]]
Sent: Wednesday, October 17, 2012 11:43 PM
To: [EMAIL PROTECTED]
Subject: Re: HBase Issue after Namenode Formating

Can you try clearing all the state in ZK as well? (stop ZK, wipe the ZK data directory, restart ZK)

----- Original Message -----
From: Pankaj Misra <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc:
Sent: Wednesday, October 17, 2012 11:09 AM
Subject: RE: HBase Issue after Namenode Formating

Hi All,

I did some more fact finding around this issue, and it seems to me that the Master is not getting initialized and hence I am not able to create table. I tried connecting via the Java client as well, which gave me the following exception. I am not sure what is causing Master to not initialize at all. I am not able to figure out, where am I going wrong, and would request help. Thanks.

java.lang.RuntimeException: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
        at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:56)
        at com.benchmark.uds.hbase.RequestGenerator.main(RequestGenerator.java:72)
Caused by: java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1350496120102/Put/vlen=15/ts=0, .META.,,1/info:serverstartcode/1350496120102/Put/vlen=8/ts=0}
  ��     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:985)
��       at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
        at org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:200)
        at org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:226)
        at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:705)
��       at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:183)
        at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:448)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:233)
        at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:247)
        at com.benchmark.uds.hbase.NativeJavaHBaseLoader.createSchema(NativeJavaHBaseLoader.java:44)

Also, the error as seen in zookeeper log for the above session is
2012-10-17 23:32:04,274 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51602 which had sessionid 0x13a6fd74b50000c
2012-10-17 23:32:05,105 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x13a6fd74b500009, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
        at java.lang.Thread.run(Thread.java:662)
2012-10-17 23:32:05,105 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:51515 which had sessionid 0x13a6fd74b500009

Thanks and Regards
Pankaj Misra
________________________________________
From: Pankaj Misra
Sent: Wednesday, October 17, 2012 10:30 PM
To: [EMAIL PROTECTED]
Subject: RE: HBase Issue after Namenode Formating

Hi All,

I have been using HBase 0.94.1 with Hadoop 0.23.1. I had the complete setup in a pseudo-distributed mode, and was running fine.

Today I had to reset and reformat the namenode and clear all the data as well to have a fresh run, but since then Hbase is not working as expected. For instance I am able to list the tables but cannot create new one, it reports that Master is initializing, as seen from the exception below

hbase(main):004:0> create 'testrun','samplecf'

ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
        at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1626)
        at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1154)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

In order to resolve this I have even tried to delete the root so that hbase can recreate it fresh. HBase is able to recreate the ROOT but this issue persists. I am also not able to connect via native java client, as it does not get reference of the Master.

Additionally I see the following error in the HBase Master logs

2012-10-17 21:15:33,04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB