Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR


Copy link to this message
-
Re: Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR
Update::

I pre split the regions and my big  data load MR worked! Thanks for your
help.

One note however, it takes time to create table with pre split regions. And
I have not figured out why.
I continue to see regions as "not deployed" for at least 20-30 minutes
after table is created with pre splits.

Only after those regions are deployed, I can start MR job.

On Thu, Nov 1, 2012 at 4:56 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:

> Hi Kevin,
>
> I was trying to pre split the table from shell, but either compression or
> splitting did not work.
>
> I tried following:
>
> create 'test1', { NAME => 'cf1', SPLITS => ['a', 'b', 'c', 'd', 'e', 'f',
> 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p' 'q', 'r', 's', 't', 'u',
> 'v', 'w', 'z'] }
> disable 'test1'
> alter 'test1' { NAME => 'cf1', COMPRESSION => 'SNAPPY' }
>
> This worked so far, however
> enable 'test1'  is perpetually stuck.
>
> So i tried building table from the JAVA API.
>
>                     // test code
>                     HTableDescriptor desc = new HTableDescriptor();
>                     desc.setName(hbaseTable.getBytes());
>                     colDesc = new HColumnDescriptor("cf1");
>
> colDesc.setCompressionType(Compression.Algorithm.SNAPPY);
>                     desc.addFamily(colDesc);
>                     byte[][] splits = new byte[23][];
>                     splits[0] = "a".getBytes();
>                      ..
>                     splits[22] = "z".getBytes();
>                     admin.createTable(desc,splits);
>
> This created the pre split table with compression on. However, when I
> started running MR over this table,
> I started getting following errors:
>
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in .META. for region t1,e,1351811844769.6a7fb0904c323917d322aef160f129cb.
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1524)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
> at com.groupon.smartdeals.mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:114)
> at com.groupon.smartdeals.mr.hbase.LoadUserCacheInHba
>
> On your earlier suggestion, I tried:
> rm /hbase/tablename
> hbck -fixMeta -fixAssignments
> restart HBase if it is still present
>
> This took away the inconsistencies, but data is gone too.
>
> Also, I have noticed that some regions of my pre split table are in "not
> deployed" state. Hbase Master web console shows following for one of the
> regions of table t1:
>
>
> t1,e,1351811844769.6a7fb0904c323917d322aef160f129cb. not deployed
>
> There are not any relevant exceptions on master or on region servers. I
> am occasionally seeing:
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException
>
> We are trying out Hbase at Groupon, and this cluster is not in production
> yet, so thankfully we can afford to wipe the data, or restart the server
> etc.
>
> Thanks,
>
> Ameya
>
>
>
>
> On Thu, Nov 1, 2012 at 12:55 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:
>
>> Ameya,
>>
>>  If your new table goes well(did you presplit this time?), then what we
>> can
>> do for the old one:
>>
>> rm /hbase/tablename
>> hbck -fixMeta -fixAssignments
>> restart HBase if it is still present
>> All should be well.
>>
>> Please let us know how it goes.
>>
>> On Thu, Nov 1, 2012 at 2:44 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:
>>
>> > Thanks Kevin & Ram. Please find my answers below: