Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> A dropped table has region still deployed on region server


Copy link to this message
-
A dropped table has region still deployed on region server
Hi All,

hbck is reporting this error:

ERROR: Region UNKNOWN_REGION on ip-4.example.com,60020,1334705737039,
key=5fe2189f32a52f3f2bce55a92ae5df9d, not on HDFS or in META but deployed
on ip-4.example.com,60020,1334705737039

Region server log :
http://pastie.org/3823850

Namenode audit log:
http://pastie.org/3823867

Master log:
http://pastie.org/3823871

>From the region server log it seems there was a hiccup during a split event
as seen here:

2189f32a52f3f2bce55a92ae5df9d.; Took too long to split the files and create
the references, aborting split
java.io.IOException: Took too long to split the files and create the
references, aborting split
>-at
org.apache.hadoop.hbase.regionserver.SplitTransaction.splitStoreFiles(SplitTransaction.java:613)
>-at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:285)
>-at
org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:450)
>-at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
>-at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>-at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>-at java.lang.Thread.run(Thread.java:619)
2012-04-19 19:32:31,690 INFO
org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROWCOL
CompoundBloomFilter metadata for 877edbbabbf74e2e94888ddb29fb1f1b
2012-04-19 19:32:31,730 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Cleaned up old
failed split transaction detritus: hdfs://
master-hadoop.example.com:8020/hbase/TestUserBTProfile/5fe2189f32a52f3f2bce55a92ae5df9d/splits
2012-04-19 19:32:31,731 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined
TestUserBTProfile,141018963342131767,1334870127822.5fe2189f32a52f3f2bce55a92ae5df9d.;
next sequenceid=1823595
2012-04-19 19:32:31,752 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback of
failed split of
TestUserBTProfile,141018963342131767,1334870127822.5fe2189f32a52f3f2bce55a92ae5df9d.

-Shrijeet
+
Shrijeet Paliwal 2012-04-20, 17:33
+
Ted Yu 2012-04-20, 18:23
+
Shrijeet Paliwal 2012-04-20, 18:36
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB