Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR


+
Ameya Kantikar 2012-11-01, 07:02
Copy link to this message
-
Re: Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR
One more thing, the Hbase table in question is neither enabled, nor
disabled:

hbase(main):006:0> is_disabled 'userTable1'
false

0 row(s) in 0.0040 seconds

hbase(main):007:0> is_enabled 'userTable1'
false

0 row(s) in 0.0040 seconds

Ameya

On Thu, Nov 1, 2012 at 12:02 AM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am trying to load lot of data (around 1.5 TB) into a single Hbase table.
> I have setup region size at 2 GB. I also
> set hbase.regionserver.handler.count at 30.
>
> When I start loading data via MR, after a while, tasks start failing with
> following error:
>
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NotServingRegionException: 1 time, servers with issues: smartdeals-hbase8-snc1.snc1:60020,
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1641)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
> at com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:83)
> at com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:33)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.j
>
> On the hbase8 machine I see following in logs:
>
> ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Error while syncing, requesting close of hlog
> java.io.IOException: Reflection
>         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:230)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1109)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1213)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1071)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:228)
>         ... 4 more
>
>
> I only have 15 map tasks each on a 10 machine cluster (total 150 map tasks entering data into Hbase table).
>
> Further, I see 2-3 regions perpetually under "Regions in Transitions" in Hbase master web console as follows:
>
> 8dcb3edee4e43faa3dbeac2db4f12274userTable1,[EMAIL PROTECTED],1351728961461.8dcb3edee4e43faa3dbeac2db4f12274. state=PENDING_OPEN, ts=Thu Nov 01 06:39:57 UTC 2012 (409s ago), server=smartdeals-hbase1-snc1.snc1,60020,1351751785514
>
>
> bb91fd0c855e60dd4159e0ad3fd52cdauserTable1,[EMAIL PROTECTED],1351728968936.bb91fd0c855e60dd4159e0ad3fd52cda. state=PENDING_OPEN, ts=Thu Nov 01 06:42:17 UTC 2012 (269s ago), server=smartdeals-hbase3-snc1.snc1,60020,1351747466016
> bd44334a11464baf85013c97d673e600userTable1,[EMAIL PROTECTED],1351728952308.bd44334a11464baf85013c97d673e600. state=PENDING_OPEN, ts=Thu Nov 01 06:42:17 UTC 2012 (269s ago), server=smartdeals-hbase1-snc1.snc1,60020,1351751785514
> ed1f7e7908fc232f10d78dd1e796a5d7userTable1,[EMAIL PROTECTED],1351728971232.ed1f7e7908fc232f10d78dd1e796a5d7. state=PENDING_OPEN, ts=Thu Nov 01 06:37:37 UTC 2012 (549s ago), server=smartdeals-hbase3-snc1.snc1,60020,1351747466016
>
>
> Note these are not going away even after 30 minutes.
>
> Further after running
>
> hbase hbck -summary I get following:
>
> Summary:
>   -ROOT- is okay.
>     Number of regions: 1
>     Deployed on:  smartdeals-hbase7-snc1.snc1,60020,1351747458782
+
Cheng Su 2012-11-01, 08:19
+
Ameya Kantikar 2012-11-01, 08:43
+
Cheng Su 2012-11-01, 09:16
+
ramkrishna vasudevan 2012-11-01, 10:20
+
Kevin Odell 2012-11-01, 13:35
+
Ameya Kantikar 2012-11-01, 19:44
+
Kevin Odell 2012-11-01, 19:55
+
Ameya Kantikar 2012-11-01, 23:56
+
Ameya Kantikar 2012-11-03, 00:10
+
Michael Segel 2012-11-01, 14:50
+
Kevin Odell 2012-11-01, 15:35
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB