Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR


Copy link to this message
-
Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR
Hi,

I am trying to load lot of data (around 1.5 TB) into a single Hbase table.
I have setup region size at 2 GB. I also
set hbase.regionserver.handler.count at 30.

When I start loading data via MR, after a while, tasks start failing with
following error:

org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: NotServingRegionException: 1 time, servers with
issues: smartdeals-hbase8-snc1.snc1:60020,
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1641)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
at com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:83)
at com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:33)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.j

On the hbase8 machine I see following in logs:

ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Error while
syncing, requesting close of hlog
java.io.IOException: Reflection
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:230)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1109)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1213)
        at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1071)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:228)
        ... 4 more
I only have 15 map tasks each on a 10 machine cluster (total 150 map
tasks entering data into Hbase table).

Further, I see 2-3 regions perpetually under "Regions in Transitions"
in Hbase master web console as follows:

8dcb3edee4e43faa3dbeac2db4f12274userTable1,[EMAIL PROTECTED],1351728961461.8dcb3edee4e43faa3dbeac2db4f12274.
state=PENDING_OPEN, ts=Thu Nov 01 06:39:57 UTC 2012 (409s ago),
server=smartdeals-hbase1-snc1.snc1,60020,1351751785514
bb91fd0c855e60dd4159e0ad3fd52cdauserTable1,[EMAIL PROTECTED],1351728968936.bb91fd0c855e60dd4159e0ad3fd52cda.
state=PENDING_OPEN, ts=Thu Nov 01 06:42:17 UTC 2012 (269s ago),
server=smartdeals-hbase3-snc1.snc1,60020,1351747466016bd44334a11464baf85013c97d673e600userTable1,[EMAIL PROTECTED],1351728952308.bd44334a11464baf85013c97d673e600.
state=PENDING_OPEN, ts=Thu Nov 01 06:42:17 UTC 2012 (269s ago),
server=smartdeals-hbase1-snc1.snc1,60020,1351751785514ed1f7e7908fc232f10d78dd1e796a5d7userTable1,[EMAIL PROTECTED],1351728971232.ed1f7e7908fc232f10d78dd1e796a5d7.
state=PENDING_OPEN, ts=Thu Nov 01 06:37:37 UTC 2012 (549s ago),
server=smartdeals-hbase3-snc1.snc1,60020,1351747466016
Note these are not going away even after 30 minutes.

Further after running

hbase hbck -summary I get following:

Summary:
  -ROOT- is okay.
    Number of regions: 1
    Deployed on:  smartdeals-hbase7-snc1.snc1,60020,1351747458782
  .META. is okay.
    Number of regions: 1
    Deployed on:  smartdeals-hbase7-snc1.snc1,60020,1351747458782
  test1 is okay.
    Number of regions: 1
    Deployed on:  smartdeals-hbase2-snc1.snc1,60020,1351747457308
  userTable1 is okay.
    Number of regions: 32
    Deployed on:  smartdeals-hbase10-snc1.snc1,60020,1351747456776
smartdeals-hbase2-snc1.snc1,60020,1351747457308
smartdeals-hbase4-snc1.snc1,60020,1351747455571
smartdeals-hbase5-snc1.snc1,60020,1351747458579
smartdeals-hbase6-snc1.snc1,60020,1351747458186
smartdeals-hbase7-snc1.snc1,60020,1351747458782
smartdeals-hbase8-snc1.snc1,60020,1351747459112
smartdeals-hbase9-snc1.snc1,60020,1351747455106
24 inconsistencies detected.
Status: INCONSISTENT

In master logs I am seeing following error:

ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed
assignment in: smartdeals-hbase3-snc1.snc1,60020,1351747466016 due to
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
Received:OPEN for the
region:userTable1,[EMAIL PROTECTED],1351728968936.bb91fd0c855e60dd4159e0ad3fd52cda.
,which we are already trying to OPEN.
 at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:2499)
       at   org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2457)
       at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)        at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
       at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Am I missing something? How do I recover from this? How do I load lot
of data via MR into Hbase Tables?
I am running under following setup:

hadoop:2.0.0-cdh4.0.1

hbase: 0.92.1-cdh4.0.1, r
Would greatly appreciate any help.
Ameya
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB