Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR


Copy link to this message
-
Re: Table in Inconsistent State; Perpetually pending region server transitions while loading lot of data into Hbase via MR
Ameya Kantikar 2012-11-01, 23:56
Hi Kevin,

I was trying to pre split the table from shell, but either compression or
splitting did not work.

I tried following:

create 'test1', { NAME => 'cf1', SPLITS => ['a', 'b', 'c', 'd', 'e', 'f',
'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p' 'q', 'r', 's', 't', 'u',
'v', 'w', 'z'] }
disable 'test1'
alter 'test1' { NAME => 'cf1', COMPRESSION => 'SNAPPY' }

This worked so far, however
enable 'test1'  is perpetually stuck.

So i tried building table from the JAVA API.

                    // test code
                    HTableDescriptor desc = new HTableDescriptor();
                    desc.setName(hbaseTable.getBytes());
                    colDesc = new HColumnDescriptor("cf1");

colDesc.setCompressionType(Compression.Algorithm.SNAPPY);
                    desc.addFamily(colDesc);
                    byte[][] splits = new byte[23][];
                    splits[0] = "a".getBytes();
                     ..
                    splits[22] = "z".getBytes();
                    admin.createTable(desc,splits);

This created the pre split table with compression on. However, when I
started running MR over this table,
I started getting following errors:

org.apache.hadoop.hbase.client.NoServerForRegionException: No server
address listed in .META. for region
t1,e,1351811844769.6a7fb0904c323917d322aef160f129cb.
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:988)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1524)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
at com.groupon.smartdeals.mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:114)
at com.groupon.smartdeals.mr.hbase.LoadUserCacheInHba

On your earlier suggestion, I tried:
rm /hbase/tablename
hbck -fixMeta -fixAssignments
restart HBase if it is still present

This took away the inconsistencies, but data is gone too.

Also, I have noticed that some regions of my pre split table are in "not
deployed" state. Hbase Master web console shows following for one of the
regions of table t1:
t1,e,1351811844769.6a7fb0904c323917d322aef160f129cb. not deployed

There are not any relevant exceptions on master or on region servers. I
am occasionally seeing:
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException

We are trying out Hbase at Groupon, and this cluster is not in production
yet, so thankfully we can afford to wipe the data, or restart the server
etc.

Thanks,

Ameya
On Thu, Nov 1, 2012 at 12:55 PM, Kevin O'dell <[EMAIL PROTECTED]>wrote:

> Ameya,
>
>  If your new table goes well(did you presplit this time?), then what we can
> do for the old one:
>
> rm /hbase/tablename
> hbck -fixMeta -fixAssignments
> restart HBase if it is still present
> All should be well.
>
> Please let us know how it goes.
>
> On Thu, Nov 1, 2012 at 2:44 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:
>
> > Thanks Kevin & Ram. Please find my answers below:
> >
> > Did you presplit your table? - NO
> >
> > You are on .92, might as well take advantage of HFilev2 and use 10GB
> region
> > sizes -
> >
> >  - I have put my region size now at 10GB and running another load in a
> > separate table, but my existing table is still in bad shape.
> >
> > Loading over MR, I am assuming puts?
> > -Yes
> >
> > Did you tune your memstore and Hlog
> > size?
> > -Not yet. I am running with whatever are the defaults.
> >
> > You aren't using a different client version or something strange like
> that
> > are you? - Nope. Its the same jar everywhere.