Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> hbase corruption - missing region files in HDFS


+
Chris Waterson 2012-12-09, 04:30
+
Kevin Odell 2012-12-09, 23:00
+
Chris Waterson 2012-12-09, 23:29
Copy link to this message
-
Re: hbase corruption - missing region files in HDFS
Chris,

Thank you for the very descriptive update.

On Sun, Dec 9, 2012 at 6:29 PM, Chris Waterson <[EMAIL PROTECTED]> wrote:

> Well, I upgraded to 0.92.2, since the version I was running on (0.92.1)
> didn't have those options for "hbck".
>
> That helped.
>
> It took me a while to realize that I had to make the root filesystem
> writable so that "hbck
> -repair" could create itself a directory.  So, once that was done, it at
> least ran through to completion.
>
> But the problem persisted in that there were blocks in META that didn't
> exist on the filesystem.  One poor region server was assigned the sad task
> of attempting to open the non-existent directory, which it slavishly
> reattempted again and again, filling its log with FileNotFoundException
> stack traces.
>
> For example,
>
> 2012-12-09 00:14:33,315 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
> of
> region=referrers,com.free-hdwallpapers.www/wallpapers/animals/mici/595718.jpg|com.free-hdwallpapers.www/wallpaper/animals/husky/270579,1354964606745.0c54fe59c58ddd6b34042ec98171bff7.
> java.io.FileNotFoundException: File does not exist:
> /hbase/referrers/2cb553c74d52ddcbf31940f6c7128c63/main/33f1fd9efb944c4e982ba719cd7dde84
> etc., etc.
>
> In particular, the directory above "/hbase/referrers/2cb553...c63" simply
> did not exist at all in HDFS.
>
> So I took matters into my own hands and created the missing
> "/hbase/referrers/2cb553...c63" directory, its subdirectory "main", and
> attempted to create a zero-length file "331fd9...e84".  This changed the
> firehose of exceptions from FileNotFoundException to CorruptHFileException.
>
> So, I wrote a small program to emit a valid, empty HFile, and proceeded to
> place these files at whatever places in HDFS that a FileNotFoundException
> was being thrown.  After creating three or four of them, the exceptions
> stopped.
>
> I then ran "hbck -repair" again, and upon completion it declared victory.
>
> Again, I suspect that I got myself into this problem because I ran a
> machine out of disk space.  It's likely that most folks are more clever
> than me, and so this problem hasn't arisen before. :)
>
>
>
>
> On Dec 9, 2012, at 3:00 PM, "Kevin O'dell" <[EMAIL PROTECTED]>
> wrote:
>
> > can you run hbase hbck -fixMeta -fixAssignments
> >
> > This should assign those region servers and fix the hole.
> >
> > On Sat, Dec 8, 2012 at 11:30 PM, Chris Waterson <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hello!  I've gotten myself into trouble where I'm missing files on HDFS
> >> that HBase thinks ought to be there.  In particular, running "hbase
> hbck"
> >> yields the below message: two regions are "not deployed on any region
> >> server" (because there is no file in HDFS for the region), and "there
> is a
> >> hole in the region chain".
> >>
> >> (FWIW, I suspect that this problem is due to a recent incident where we
> >> ran the cluster out of disk space.)
> >>
> >> I'm running 0.92.1, and have been staggering around trying to figure out
> >> what procedure I ought to use to correct the problem, but my Google-fu
> is
> >> too poor to have yielded results.  Any pointers would be appreciated!
> >>
> >> thanks,
> >> chris
> >>
> >>
> >>
> >>
> >> ERROR: Region
> >>
> referrers,com.free-hdwallpapers.www/wallpapers/animals/mici/595718.jpg|com.free-hdwallpapers.www/wallpaper/animals/husky/270579,1354964606745.0c54fe59c58ddd6b34042ec98171bff7.
> >> not deployed on any region server.
> >> ERROR: Region
> >>
> referrers,com.free-hdwallpapers.www/wallpapers/anime/mici/78285.jpg|com.free-hdwallpapers.www/wallpaper/anime/wolf-furry/90641,1354964606745.d2451e8db0f2b9546cc42c6d260a2ab8.
> >> not deployed on any region server.
> >> ERROR: There is a hole in the region chain between
> >>
> com.free-hdwallpapers.www/wallpapers/animals/mici/595718.jpg|com.free-hdwallpapers.www/wallpaper/animals/husky/270579
> >> and
> >>
> com.free-hdwallpapers.www/wallpapers/entertainment/mici/11840.jpg|com.free-hdwallpapers.www/wallpaper/entertainment/new-moon-bella-and-edward/12951.
Kevin O'Dell
Customer Operations Engineer, Cloudera
+
Tom Brown 2012-12-10, 18:07
+
Chris Waterson 2012-12-10, 23:03
+
Kyle McGovern 2012-12-12, 05:26
+
Kyle McGovern 2012-12-10, 03:09
+
lars hofhansl 2012-12-11, 05:10