Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Problem When Prefetch META table


Copy link to this message
-
Re: Problem When Prefetch META table
Hey Andrea,

  That is very weird, I hit this similar extremely strange one off issue
that I could never reproduce.  After dropping a table doing X amount of
inserts, I would receive the same error and my META table would be
corrupted.  What I did to correct this issue was to sideline the META table
and then use offlineMetaRepair to rebuild it, and then I never ran into it
again.  It was quite strange and I am sorry I don't have more to offer.
On Tue, Dec 10, 2013 at 8:34 AM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi Andrea,
>
> Which version of HBase are you testing with?
>
> JM
>
>
> 2013/12/10 Andrea <[EMAIL PROTECTED]>
>
> > Hi, I encounter this problem:
> >
> > 13/12/10 13:25:15 WARN
> client.HConnectionManager$HConnectionImplementation:
> > Encountered problems when prefetch META table:
> > java.io.IOException: HRegionInfo was null or empty in Meta for usertable,
> > row=usertable,user4300292263938371081,99999999999999
> >         at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:170)
> >         at
> >
> org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
> >         at
> >
> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
> >         at
> >
> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager
> > .java:360)
> >         at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
> >         at
> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > prefetchRegionCache(HConnectionManager.java:918)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > locateRegionInMeta(HConnectionManager.java:973)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > locateRegion(HConnectionManager.java:860)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > locateRegion(HConnectionManager.java:817)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > processBatchCallback(HConnectionManager.java:1507)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> > processBatch(HConnectionManager.java:1392)
> >         at
> > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:918)
> >         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:774)
> >         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:749)
> >         at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:328)
> >         at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
> >         at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
> >         at
> > com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
> >         at com.yahoo.ycsb.ClientThread.run(Client.java:269)
> >
> > It will happen when, on a netowrk with one zookeeper, one master and one
> > regionserver I try to load a lot of data. This is the number of
> operations
> > before the error:
> >
> > 200 sec: 888708 operations; 7767.62 current ops/sec; [UPDATE
> > AverageLatency(us)=27089963] [INSERT AverageLatency(us)=9738.99] [CLEANUP
> > AverageLatency(us)=27089976]
> >  210 sec: 893742 operations; 502.7 current ops/sec; [UPDATE
> > AverageLatency(us)=14887298.5] [INSERT AverageLatency(us)=6937.27]
> [CLEANUP
> > AverageLatency(us)=14887312.5]
> >  221 sec: 928277 operations; 3226.97 current ops/sec; [UPDATE
> > AverageLatency(us)=27006074] [INSERT AverageLatency(us)=8529.21] [CLEANUP
> > AverageLatency(us)=27006089]
> >  231 sec: 930079 operations; 180.18 current ops/sec;  [INSERT
> > AverageLatency(us)=457.08]
> > 13/12/10 13:25:13 WARN

Kevin O'Dell
Systems Engineer, Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB