Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Inconsistent Table HBCK


+
Jay Talreja 2013-05-21, 11:03
+
Jean-Marc Spaggiari 2013-05-22, 03:12
+
Jay Talreja 2013-05-22, 10:52
+
Jean-Marc Spaggiari 2013-05-22, 14:56
Copy link to this message
-
RE: Inconsistent Table HBCK
Hi, Jean

What is the jira #?

Thanks
Tian-Ying
-----Original Message-----
From: Jean-Marc Spaggiari [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, May 22, 2013 7:57 AM
To: [EMAIL PROTECTED]
Subject: Re: Inconsistent Table HBCK

Thanks for the feedback Jay.

I helped someone who faced the same issue recently. Might deserve a fix...
(and so a JIRA with details)

Also, I would recommend you to migrate to a sooner 0.94.x version.

JM

2013/5/22 Jay Talreja <[EMAIL PROTECTED]>

> *0.94.0*
>
> The issue (I think) was related to a region split that didn't happen
> cleanly. As a result there were references to daughter region present
> in HDFS.
> I believe the -fixSplitParents would have taken care of this but it is
> not available in 0.94.0
>
> I manually deleted reference files from HDFS and was able to bring the
> table back to consistent state.
>
> Thanks,
> Jay
>
>
> On 5/21/13 11:12 PM, Jean-Marc Spaggiari wrote:
>
>> Hi Jay,
>>
>> Which 0.94 version are you running? 0.94.0? Or 0.94.7?
>>
>>
>> JM
>>
>> 2013/5/21 Jay Talreja<[EMAIL PROTECTED]**>
>>
>>  One of our production clusters had several region server failures.
>> As a
>>> result one of the tables is in an inconsistent state as reported by hbck.
>>> We have tried using hbck repair commands but none seem to work.
>>> There is one region that is stuck in a forever pending open state.
>>> The error reported in RS log is about a StoreFile not found. But
>>> what is really strange is that the store file that is reported as
>>> missing does not even belong to the region being opened.
>>>
>>> We tried to manually create a directory in HDFS and copy the missing
>>> file but it causes hbck to report about a region in HDFS but not in Meta.
>>>
>>> There 4 inconsistencies currently.
>>>
>>> ERROR: Region { meta =>  <tableName>,I.1521_D.****
>>> 1361689200_9,1369099149747.
>>> ****2123fc70fac804cd8d48ea4494cc81****84., hdfs =>
>>>  hdfs://host:8020/hbase/**
>>> tableName/****2123fc70fac804cd8d48ea4494cc81****84, deployed =>   } not
>>>
>>> deployed on any region server.
>>> ERROR: Region { meta =>  null, hdfs =>  hdfs://hostname:8020/hbase/**
>>> tableName/****450ed30b410e9d6d54ac53099039cb****28, deployed =>   } on
>>> HDFS,
>>>
>>> but not listed in META or deployed on any region server
>>> 13/05/21 10:51:11 DEBUG util.HBaseFsck: There are 1769 region info
>>> entries
>>> ERROR: There is a hole in the region chain between
>>> I.1521_D.1361689200_9 and I.1521_D.1362150000_8.  You need to create
>>> a new .regioninfo and region dir in hdfs to plug the hole.
>>> ERROR: There is a hole in the region chain between
>>> I.1_D.1368392400_9 and I.2020_D.1338948000_2.  You need to create a
>>> new .regioninfo and region dir in hdfs to plug the hole.
>>> ERROR: Found inconsistency in table<tableName>
>>>
>>> We are running Hbase 0.94 (Apache) on Hadoop 1.0.3
>>>
>>> At this stage, we are stuck and are looking for help ! The cluster
>>> is in an unbalanced state and region servers frequently keep dying.
>>>
>>> Thanks,
>>> Jay
>>>
>>>
>>>
>
+
Jean-Marc Spaggiari 2013-05-28, 14:27
+
Tianying Chang 2013-05-29, 16:50