Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR


Copy link to this message
-
Re: Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR
Update::

I pre split the regions and my big  data load MR worked! Thanks for your
help.

One note however, it takes time to create table with pre split regions. And
I have not figured out why.
I continue to see regions as "not deployed" for at least 20-30 minutes
after table is created with pre splits.

Only after those regions are deployed, I can start MR job.

On Thu, Nov 1, 2012 at 4:56 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:

> Hi Kevin,
>
> I was trying to pre split the table from shell, but either compression or
> splitting did not work.
>
> I tried following:
>
> create 'test1', { NAME => 'cf1', SPLITS => ['a', 'b', 'c', 'd', 'e', 'f',
> 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p' 'q', 'r', 's', 't', 'u',
> 'v', 'w', 'z'] }
> disable 'test1'
> alter 'test1' { NAME => 'cf1', COMPRESSION => 'SNAPPY' }
>
> This worked so far, however
> enable 'test1'  is perpetually stuck.
>
> So i tried building table from the JAVA API.
>
>                     // test code
>                     HTableDescriptor desc = new HTableDescriptor();
>                     desc.setName(hbaseTable.getBytes());
>                     colDesc = new HColumnDescriptor("cf1");
>
> colDesc.setCompressionType(Compression.Algorithm.SNAPPY);
>                     desc.addFamily(colDesc);
>                     byte[][] splits = new byte[23][];
>                     splits[0] = "a".getBytes();
>                      ..
>                     splits[22] = "z".getBytes();
>                     admin.createTable(desc,splits);
>
> This created the pre split table with compression on. However, when I
> started running MR over this table,
> I started getting following errors:
>
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in .META. for region t1,e,1351811844769.6a7fb0904c323917d322aef160f129cb.
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1524)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
> at com.groupon.smartdeals.mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:114)
> at com.groupon.smartdeals.mr.hbase.LoadUserCacheInHba
>
> On your earlier suggestion, I tried:
> rm /hbase/tablename
> hbck -fixMeta -fixAssignments
> restart HBase if it is still present
>
> This took away the inconsistencies, but data is gone too.
>
> Also, I have noticed that some regions of my pre split table are in "not
> deployed" state. Hbase Master web console shows following for one of the
> regions of table t1:
>
>
> t1,e,1351811844769.6a7fb0904c323917d322aef160f129cb. not deployed
>
> There are not any relevant exceptions on master or on region servers. I
> am occasionally seeing:
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException
>
> We are trying out Hbase at Groupon, and this cluster is not in production
> yet, so thankfully we can afford to wipe the data, or restart the server
> etc.
>
> Thanks,
>
> Ameya
>
>
>
>
> On Thu, Nov 1, 2012 at 12:55 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:
>
>> Ameya,
>>
>>  If your new table goes well(did you presplit this time?), then what we
>> can
>> do for the old one:
>>
>> rm /hbase/tablename
>> hbck -fixMeta -fixAssignments
>> restart HBase if it is still present
>> All should be well.
>>
>> Please let us know how it goes.
>>
>> On Thu, Nov 1, 2012 at 2:44 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:
>>
>> > Thanks Kevin & Ram. Please find my answers below:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB