Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Excessive .META scans


Copy link to this message
-
Re: Excessive .META scans
Just patched 6870 and it immediately fixed the problem !
On Tue, Jul 30, 2013 at 12:57 PM, Stack <[EMAIL PROTECTED]> wrote:

> Try turning off
>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#setRegionCachePrefetch(byte[]
> ,
> boolean)
>
> St.Ack
>
>
> On Tue, Jul 30, 2013 at 11:27 AM, Varun Sharma <[EMAIL PROTECTED]>
> wrote:
>
> > JD, its a big problem. The region server holding .META has 2X the network
> > traffic and 2X the cpu load, I can easily spot the region server holding
> > .META. by just looking at the ganglia graphs of the region servers side
> by
> > side - I don't need to go the master console. So we can't scale up the
> > cluster or add more load since its bottlenecked on this one region
> server.
> >
> > Thanks Nicholas for the pointer, its seems quite probable that this is
> the
> > issue - it was fixed with 0.94.8 so we don't have it. I will give it a
> > shot.
> >
> >
> > On Mon, Jul 29, 2013 at 10:43 AM, Nicolas Liochon <[EMAIL PROTECTED]>
> > wrote:
> >
> > > It could be HBASE-6870?
> > >
> > >
> > > On Mon, Jul 29, 2013 at 7:37 PM, Jean-Daniel Cryans <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Can you tell who's doing it? You could enable IPC debug for a few
> secs
> > > > to see who's coming in with scans.
> > > >
> > > > You could also try to disable pre-fetching, set
> > > > hbase.client.prefetch.limit to 0
> > > >
> > > > Also, is it even causing a problem or you're just worried it might
> > > > since it doesn't look "normal"?
> > > >
> > > > J-D
> > > >
> > > > On Mon, Jul 29, 2013 at 10:32 AM, Varun Sharma <[EMAIL PROTECTED]>
> > > > wrote:
> > > > > Hi folks,
> > > > >
> > > > > We are seeing an issue with hbase 0.94.3 on CDH 4.2.0 with
> excessive
> > > > .META.
> > > > > reads...
> > > > >
> > > > > In the steady state where there are no client crashes and there are
> > no
> > > > > region server crashes/region movement, the server holding .META. is
> > > > serving
> > > > > an incredibly large # of read requests on the .META. table.
> > > > >
> > > > > From my understanding, in the steady state, region locations should
> > be
> > > > > indefinitely cached in the client. The client is running a work
> load
> > of
> > > > > multiput(s), puts, gets and coprocessor calls.
> > > > >
> > > > > Thanks
> > > > > Varun
> > > >
> > >
> >
>