Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> major compaction and regionserver failure...


+
Andy Sautins 2011-12-10, 18:37
+
Mikhail Bautin 2011-12-10, 20:49
+
Andy Sautins 2011-12-10, 22:25
Copy link to this message
-
Re: major compaction and regionserver failure...
Hi Andy,

We don't persist manual compaction requests across server restarts, so if
e.g. a major compaction was requested manually and the regionserver
crashed, the new regionserver that picks up that region will not know about
the major compaction request. For automatic compactions, however,
compaction criteria are the same on the old and the new region server, so
we would expect the new server to figure out that a compaction is necessary
and start it again from the beginning if the old regionserver died in the
middle of a compaction.

Thanks,
--Mikhail

On Sat, Dec 10, 2011 at 2:25 PM, Andy Sautins
<[EMAIL PROTECTED]>wrote:

>
>  Mikhail,
>
>  Thanks for the response.  So to summarize compaction requests in the
> compaction queue of the failed regionserver are lost and not picked up when
> the regions are picked up by another regionserver.  So if we lose a
> regionserver during a major compaction the regions that had not yet been
> major compacted on the failed regionserver will not be major compacted as
> part of that major compaction request, correct?
>
>  Thanks for the insight.
>
>   Andy
>
> -----Original Message-----
> From: Mikhail Bautin [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, December 10, 2011 1:50 PM
> To: [EMAIL PROTECTED]
> Subject: Re: major compaction and regionserver failure...
>
> Hi Andy,
>
> Compaction queues are not persisted between regionserver restarts, and the
> results of an incomplete compaction are discarded. Compactions write into
> an HFile in a temporary location and only move it to the region/CF
> directory in case of success (at least, this is how it works in trunk).
>
> Thanks,
> --Mikhail
>
> On Saturday, December 10, 2011, Andy Sautins wrote:
>
> >
> >   I was wondering if someone could tell me what happens if a
> > regionserver fails during major compaction.  Are the entries that were
> > in the compaction queue for the failed regionserver migrated along
> > with the regions to another server or are those requests for major
> > compaction effectively lost for that run of major compaction?  I guess
> > what I'm really asking is do I still get a full major compaction of
> > all regions or do I lose the major compactions of the regions that
> > were in the queue on the failed regionserver.
> >
> >   Any insight would be much appreciated.   We are currently running HBase
> > 0.90.3-cdh3u1.
> >
> >
> >
>
+
Andy Sautins 2011-12-11, 19:40
+
Nicolas Spiegelberg 2011-12-11, 20:46
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB