Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - 0.94.2 failing regionservers


Copy link to this message
-
Re: 0.94.2 failing regionservers
Bryan Beaudreault 2013-03-04, 17:26
We upgraded from CDH3 which is HBase 0.90.  Looking around we do have some
v2 format HFiles, just not entirely there yet.  I'll try to get them all
converted.
On Mon, Mar 4, 2013 at 12:23 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> In hindsight, this should have helped:
>
> http://hbase.apache.org/book.html#upgrade0.94
>
> I would suggest upgrading existing HFile v1 files to v2 format.
>
> Cheers
>
> On Mon, Mar 4, 2013 at 9:20 AM, Bryan Beaudreault
> <[EMAIL PROTECTED]>wrote:
>
> > Interesting.
> >
> > We upgraded by creating entirely new clusters with the CDH4.2 software
> > installed onto it. Then we distributed copied the /hbase directory to the
> > new clusters.  We haven't run a major compaction since then, so that
> could
> > be the reason there are still v1 HFiles.  This migration took place just
> a
> > couple days ago.
> >
> >
> > On Mon, Mar 4, 2013 at 12:04 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > In the stack trace, I see HFileReaderV1.java
> > > I would expect your store files to be upgraded to HFile v2 after
> cluster
> > > upgrade.
> > >
> > > Can you tell us more about how you upgraded your clusters ?
> > >
> > > Thanks
> > >
> > > On Mon, Mar 4, 2013 at 9:00 AM, Bryan Beaudreault
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > We recently upgraded multiple clusters to CDH 4.2, which comes with
> > hbase
> > > > 0.94.2  Since then We've seen region servers die periodically in a
> way
> > I
> > > > never saw before on CDH3.
> > > >
> > > > Here are the exceptions:
> > > >
> > > > First I see a slew of these: http://pastebin.com/WqSwMzuZ
> > > >
> > > > Then the regionserver starts closing all its regions, after throwing
> > this
> > > > exception: http://pastebin.com/396wX3iw
> > > >
> > > > This has now happened on multiple servers across multiple clusters,
> all
> > > cdh
> > > > 4.2.
> > > >
> > > > Any thoughts?  During the time our NameNodes seem to be doing fine
> and
> > I
> > > > don't see any issue on our datanodes either.
> > > >
> > >
> >
>