Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - A dropped table has region still deployed on region server


+
Shrijeet Paliwal 2012-04-20, 17:26
+
Shrijeet Paliwal 2012-04-20, 17:33
+
Ted Yu 2012-04-20, 18:23
Copy link to this message
-
Re: A dropped table has region still deployed on region server
Shrijeet Paliwal 2012-04-20, 18:36
What happened in region server :

2012-04-19 19:32:31,752 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback of
failed split of
TestUserBTProfile,141018963342131767,1334870127822.5fe2189f32a52f3f2bce55a92ae5df9d.
2012-04-19 19:50:10,392 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing
.META.,,1.1028785192
2012-04-19 19:50:10,529 INFO org.apache.hadoop.hbase.regionserver.Store:
Added hdfs://
master-hadoop.example.com:8020/hbase/.META./1028785192/info/270b515bf7d34ab88552f0e244419a0d,
entries=823, sequenceid=1825092, memsize=243.3k, filesize=102.2k
2012-04-19 19:50:10,535 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~483.5k/495080, currentsize=0.0/0 for region
.META.,,1.1028785192 in 88ms, sequenceid=1825092, compaction requested=false
2012-04-19 19:50:10,614 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on info in region .META.,,1.1028785192
2012-04-19 19:50:10,615 INFO org.apache.hadoop.hbase.regionserver.Store:
Starting compaction of 2 file(s) in info of .META.,,1.1028785192 into
tmpdir=hdfs://master-hadoop.example.com:8020/hbase/.META./1028785192/.tmp,
seqid=1825092, totalSize=541.0k
2012-04-19 19:50:10,813 INFO org.apache.hadoop.hbase.regionserver.Store:
Renaming compacted file at hdfs://
master-hadoop.example.com:8020/hbase/.META./1028785192/.tmp/5f7bea0f8f8142109c0ef41ada5a509eto
hdfs://
master-hadoop.example.com:8020/hbase/.META./1028785192/info/5f7bea0f8f8142109c0ef41ada5a509e2012-04-1919:50:10,927
INFO org.apache.hadoop.hbase.regionserver.Store: Completed
major compaction of 2 file(s) in info of .META.,,1.1028785192 into
5f7bea0f8f8142109c0ef41ada5a509e, size=442.0k; total size for store is
442.0k2012-04-19 19:50:10,927 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
completed compaction: regionName=.META.,,1.1028785192, storeName=info,
fileCount=2, fileSize=541.0k, priority=1, time=6327679323413783;
duration=0sec2012-04-19 20:04:14,620 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region
userprofile,671599294440030433,1334171984301.075898f82c75fa049e7bfb88b7ca567d.
due to global heap pressure

On Fri, Apr 20, 2012 at 11:23 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> bq. Successful rollback of failed split of
> Can you look for 'TestUserBTProfile,141018963342131767' in region / master
> logs to see what happened next ?
>
> Thanks
>
> On Fri, Apr 20, 2012 at 10:33 AM, Shrijeet Paliwal
> <[EMAIL PROTECTED]>wrote:
>
> > Here are the version details:
> >
> > Hadoop : CDH3u0
> > HBase: 0.92.1 (last commit sha :
> 355e14491c78b56d9f2e270e9995d666d4ce6767)
> >
> > On Fri, Apr 20, 2012 at 10:26 AM, Shrijeet Paliwal
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Hi All,
> > >
> > > hbck is reporting this error:
> > >
> > > ERROR: Region UNKNOWN_REGION on ip-4.example.com,60020,1334705737039,
> > > key=5fe2189f32a52f3f2bce55a92ae5df9d, not on HDFS or in META but
> deployed
> > > on ip-4.example.com,60020,1334705737039
> > >
> > > Region server log :
> > > http://pastie.org/3823850
> > >
> > > Namenode audit log:
> > > http://pastie.org/3823867
> > >
> > > Master log:
> > > http://pastie.org/3823871
> > >
> > > From the region server log it seems there was a hiccup during a split
> > > event as seen here:
> > >
> > > 2189f32a52f3f2bce55a92ae5df9d.; Took too long to split the files and
> > > create the references, aborting split
> > > java.io.IOException: Took too long to split the files and create the
> > > references, aborting split
> > > >-at
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitTransaction.splitStoreFiles(SplitTransaction.java:613)
> > > >-at
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:285)
> > > >-at
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:450)
> > > >-at
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> > > >-at