Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR


Copy link to this message
-
Re: Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR
Ameya Kantikar 2012-11-01, 07:29
One more thing, the Hbase table in question is neither enabled, nor
disabled:

hbase(main):006:0> is_disabled 'userTable1'
false

0 row(s) in 0.0040 seconds

hbase(main):007:0> is_enabled 'userTable1'
false

0 row(s) in 0.0040 seconds

Ameya

On Thu, Nov 1, 2012 at 12:02 AM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am trying to load lot of data (around 1.5 TB) into a single Hbase table.
> I have setup region size at 2 GB. I also
> set hbase.regionserver.handler.count at 30.
>
> When I start loading data via MR, after a while, tasks start failing with
> following error:
>
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NotServingRegionException: 1 time, servers with issues: smartdeals-hbase8-snc1.snc1:60020,
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1641)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
> at com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:83)
> at com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:33)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.j
>
> On the hbase8 machine I see following in logs:
>
> ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Error while syncing, requesting close of hlog
> java.io.IOException: Reflection
>         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:230)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1109)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1213)
>         at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1071)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:228)
>         ... 4 more
>
>
> I only have 15 map tasks each on a 10 machine cluster (total 150 map tasks entering data into Hbase table).
>
> Further, I see 2-3 regions perpetually under "Regions in Transitions" in Hbase master web console as follows:
>
> 8dcb3edee4e43faa3dbeac2db4f12274userTable1,[EMAIL PROTECTED],1351728961461.8dcb3edee4e43faa3dbeac2db4f12274. state=PENDING_OPEN, ts=Thu Nov 01 06:39:57 UTC 2012 (409s ago), server=smartdeals-hbase1-snc1.snc1,60020,1351751785514
>
>
> bb91fd0c855e60dd4159e0ad3fd52cdauserTable1,[EMAIL PROTECTED],1351728968936.bb91fd0c855e60dd4159e0ad3fd52cda. state=PENDING_OPEN, ts=Thu Nov 01 06:42:17 UTC 2012 (269s ago), server=smartdeals-hbase3-snc1.snc1,60020,1351747466016
> bd44334a11464baf85013c97d673e600userTable1,[EMAIL PROTECTED],1351728952308.bd44334a11464baf85013c97d673e600. state=PENDING_OPEN, ts=Thu Nov 01 06:42:17 UTC 2012 (269s ago), server=smartdeals-hbase1-snc1.snc1,60020,1351751785514
> ed1f7e7908fc232f10d78dd1e796a5d7userTable1,[EMAIL PROTECTED],1351728971232.ed1f7e7908fc232f10d78dd1e796a5d7. state=PENDING_OPEN, ts=Thu Nov 01 06:37:37 UTC 2012 (549s ago), server=smartdeals-hbase3-snc1.snc1,60020,1351747466016
>
>
> Note these are not going away even after 30 minutes.
>
> Further after running
>
> hbase hbck -summary I get following:
>
> Summary:
>   -ROOT- is okay.
>     Number of regions: 1
>     Deployed on:  smartdeals-hbase7-snc1.snc1,60020,1351747458782