Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Eternal RIT problem when RS tries to access wrong region-folder on HDFS


+
Dimitri Goldin 2013-05-02, 14:17
+
Kevin Odell 2013-05-03, 12:57
+
Dimitri Goldin 2013-05-03, 14:34
Copy link to this message
-
Re: Eternal RIT problem when RS tries to access wrong region-folder on HDFS
Dimitri,

This was the correct approach, if the directory /hbase/documents/
5b9c16898a371de58f31f0bdf86b1f**8b did not exist, then it was a smart move
to get rid of the pointers to it.  I don't think we have a JIRA for this
yet...BUT we really need one.  Can you please file one?  I think there has
been a change in the code that has caused our point of no return to be in
the wrong spot.  I am glad you got it working, you probably do not need to
keep the backup region any longer.

  You can most likely remove the region that does not have any store files
and only a .recovered.edits.  HBCK should pick this up as an orphan if you
run it.

-Kevin
On Fri, May 3, 2013 at 10:34 AM, Dimitri Goldin
<[EMAIL PROTECTED]>wrote:

> Hi Kevin,
>
>
> On 05/03/2013 02:57 PM, Kevin O'dell wrote:
> > That is interesting.  I have seen this before, can you please send a
> > hadoop fs -lsr /hbase/documents?  This is going to be caused by a bad
> > split.  I will let you know what files you need to delete to safely
> > recover from this error.
>
> Thanks for the reply. Earlier today I also determined that it has to
> do with a failed region-split and already tried to solve it
> on my own.
>
> I found a total of three reference files in the folder and two hfiles.
> Unfortunately documents contains more than 5k regions, so it seems a
> little impractical to send the listing to the list. Please let me know
> if you'd still like to see it and I will send it to you directly.
>
> original contents of /hbase/documents/**79c619508659018ff3ef0887611eb8**
> f7/d*:
> => 0707b1ec4c6b41cf9174e0d2a1785f**e9.**5b9c16898a371de58f31f0bdf86b1f**8b
> 47511faae81b4452afd3ca206e2834**6f.**5b9c16898a371de58f31f0bdf86b1f**8b
> 4f01ecd052ce464d81e79a62ea227d**6b (116MB)
> 4f01ecd052ce464d81e79a62ea227d**6b.**5b9c16898a371de58f31f0bdf86b1f**8b
> eb7dbb09701d4353be24ca82481c4a**7e (951MB)
> =>
> * d is the only Columnfamily
>
> Additionally, there was an 'almost empty' recovered.edits referencing
> the old parent region and containing only a CACHEFLUSH.
>
> As mentioned, '**5b9c16898a371de58f31f0bdf86b1f**8b' did not exist anymore
> ,.tmp was empty and .META. entry did not contain any splitA/splitB
> columns, so I backed up the original region folder, removed the
> reference files and kept 4f01ecd052ce464d81e79a62ea227d**6b
> and eb7dbb09701d4353be24ca82481c4a**7e for now to get the table working
> again.
>
> I am still trying to locate log entries from the split, but haven't
> found them yet.
>
> Do you think this was an appropriate measure? Please let me know if
> you had a different approach in mind and I'll see if I can use the
> backed-up region. Also, any ideas under which circumstances this
> might occur/is there a JIRA I can follow and maybe try to contribute
> observations from logs?
>
> Thanks a lot,
>
>         Dimitry
> --
> ------------------------------**----
> Dimitry Goldin
> Software Developer
>
> Neofonie GmbH
> Robert-Koch-Platz 4
> 10115 Berlin
>
> T: +49 30 246 27
>
> [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> http://www.neofonie.de
>
> Handelsregister
> Berlin-Charlottenburg: HRB 67460
>
> Geschäftsführung:
> Thomas Kitlitschko
>

--
Kevin O'Dell
Systems Engineer, Cloudera
+
Dimitri Goldin 2013-05-07, 15:57
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB