Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> hbase corruption - missing region files in HDFS


+
Chris Waterson 2012-12-09, 04:30
+
Kevin Odell 2012-12-09, 23:00
+
Chris Waterson 2012-12-09, 23:29
Copy link to this message
-
Re: hbase corruption - missing region files in HDFS
Chris,

Thank you for the very descriptive update.

On Sun, Dec 9, 2012 at 6:29 PM, Chris Waterson <[EMAIL PROTECTED]> wrote:

> Well, I upgraded to 0.92.2, since the version I was running on (0.92.1)
> didn't have those options for "hbck".
>
> That helped.
>
> It took me a while to realize that I had to make the root filesystem
> writable so that "hbck
> -repair" could create itself a directory.  So, once that was done, it at
> least ran through to completion.
>
> But the problem persisted in that there were blocks in META that didn't
> exist on the filesystem.  One poor region server was assigned the sad task
> of attempting to open the non-existent directory, which it slavishly
> reattempted again and again, filling its log with FileNotFoundException
> stack traces.
>
> For example,
>
> 2012-12-09 00:14:33,315 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
> of
> region=referrers,com.free-hdwallpapers.www/wallpapers/animals/mici/595718.jpg|com.free-hdwallpapers.www/wallpaper/animals/husky/270579,1354964606745.0c54fe59c58ddd6b34042ec98171bff7.
> java.io.FileNotFoundException: File does not exist:
> /hbase/referrers/2cb553c74d52ddcbf31940f6c7128c63/main/33f1fd9efb944c4e982ba719cd7dde84
> etc., etc.
>
> In particular, the directory above "/hbase/referrers/2cb553...c63" simply
> did not exist at all in HDFS.
>
> So I took matters into my own hands and created the missing
> "/hbase/referrers/2cb553...c63" directory, its subdirectory "main", and
> attempted to create a zero-length file "331fd9...e84".  This changed the
> firehose of exceptions from FileNotFoundException to CorruptHFileException.
>
> So, I wrote a small program to emit a valid, empty HFile, and proceeded to
> place these files at whatever places in HDFS that a FileNotFoundException
> was being thrown.  After creating three or four of them, the exceptions
> stopped.
>
> I then ran "hbck -repair" again, and upon completion it declared victory.
>
> Again, I suspect that I got myself into this problem because I ran a
> machine out of disk space.  It's likely that most folks are more clever
> than me, and so this problem hasn't arisen before. :)
>
>
>
>
> On Dec 9, 2012, at 3:00 PM, "Kevin O'dell" <[EMAIL PROTECTED]>
> wrote:
>
> > can you run hbase hbck -fixMeta -fixAssignments
> >
> > This should assign those region servers and fix the hole.
> >
> > On Sat, Dec 8, 2012 at 11:30 PM, Chris Waterson <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hello!  I've gotten myself into trouble where I'm missing files on HDFS
> >> that HBase thinks ought to be there.  In particular, running "hbase
> hbck"
> >> yields the below message: two regions are "not deployed on any region
> >> server" (because there is no file in HDFS for the region), and "there
> is a
> >> hole in the region chain".
> >>
> >> (FWIW, I suspect that this problem is due to a recent incident where we
> >> ran the cluster out of disk space.)
> >>
> >> I'm running 0.92.1, and have been staggering around trying to figure out
> >> what procedure I ought to use to correct the problem, but my Google-fu
> is
> >> too poor to have yielded results.  Any pointers would be appreciated!
> >>
> >> thanks,
> >> chris
> >>
> >>
> >>
> >>
> >> ERROR: Region
> >>
> referrers,com.free-hdwallpapers.www/wallpapers/animals/mici/595718.jpg|com.free-hdwallpapers.www/wallpaper/animals/husky/270579,1354964606745.0c54fe59c58ddd6b34042ec98171bff7.
> >> not deployed on any region server.
> >> ERROR: Region
> >>
> referrers,com.free-hdwallpapers.www/wallpapers/anime/mici/78285.jpg|com.free-hdwallpapers.www/wallpaper/anime/wolf-furry/90641,1354964606745.d2451e8db0f2b9546cc42c6d260a2ab8.
> >> not deployed on any region server.
> >> ERROR: There is a hole in the region chain between
> >>
> com.free-hdwallpapers.www/wallpapers/animals/mici/595718.jpg|com.free-hdwallpapers.www/wallpaper/animals/husky/270579
> >> and
> >>
> com.free-hdwallpapers.www/wallpapers/entertainment/mici/11840.jpg|com.free-hdwallpapers.www/wallpaper/entertainment/new-moon-bella-and-edward/12951.
Kevin O'Dell
Customer Operations Engineer, Cloudera
+
Tom Brown 2012-12-10, 18:07
+
Chris Waterson 2012-12-10, 23:03
+
Kyle McGovern 2012-12-12, 05:26
+
Kyle McGovern 2012-12-10, 03:09
+
lars hofhansl 2012-12-11, 05:10
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB