Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - more regionservers does not improve performance


Copy link to this message
-
Re: more regionservers does not improve performance
Jonathan Bishop 2012-10-15, 02:42
Thanks Matt,

I have 10 regions per regionserver (100 splits over 10 regionservers), and
yes they all seem to almost stop at the same time. I'll try splitting the
table into fewer regions as you suggest.

Where do I set the memstore flush size? Sorry pretty new to this.

Jon

On Sun, Oct 14, 2012 at 5:48 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:

> It could be network bound, especially if you have decently size values
> (~500B+).  HBase can be rough on the network because each value travels
> from client to regionserver, and then makes 2 additional network hops in
> the WAL, and then an additional 2 hops in the memstore flush, plus ongoing
> compactions.  After disabling the WAL, enabling GZIP compression on the
> table can cut down on the flush/compaction impact if your data is
> compressible.
>
> How long are your row keys and values, and how many cells do you have per
> row?  Longer keys would point towards internal limitations in hbase
> (locking, cpu usage, etc), while longer values indicate network and disk
> limitations.
>
> Another consideration is that your workload may be too even and is not
> given enough time to find steady state.  If you have 12-25 regions per
> server and your workload is perfectly randomized, then all regions will hit
> the memstore flush size simultaneously which triggers 12-25 memstore
> flushes at the same time.  The memstore flusher may be single threaded (i
> forget), so you are suddenly hitting the blocking storefile limit which
> could explain the pauses you are seeing.  You could try reducing the number
> of regions to ~4/server.  And make sure your memstore flush size is at
> least 256M.
>
> Matt
>
>
> On Sun, Oct 14, 2012 at 8:48 AM, Jonathan Bishop <[EMAIL PROTECTED]
> >wrote:
>
> > Matt,
> >
> > Yes, I did. What I observed is that the map job proceeds about 3-4x
> faster
> > for  a while. But then I observed long pauses partway through the job,
> and
> > overall run time was only reduced only modestly, way from 50 minutes to
> 40
> > minutes.
> >
> > Just to summarize the issue, my mapper jobs seem to scale nicely. This is
> > expected as my dfs block size is small enough to create over 500 tasks,
> and
> > I have a max of 40 mappers running.
> >
> > But when I include puts to hbase in my job, then I see a 4-6x slowdown
> > which does not respond to an increasing number of regionservers.
> >
> > My current best guess is that there is a network bottleneck in getting
> the
> > puts produced by the mappers to the appropriate regionservers, as I
> assume
> > that once the puts are received by the regionservers that they can all
> > operate in parallel without slowing each other down.
> >
> > Again, I am on grid which is used by many others, and the machines in my
> > cluster are not dedicated to my job. I am mainly looking at scalability
> > trends when running with various numbers of regionservers.
> >
> > Jon
> >
> > On Sat, Oct 13, 2012 at 10:37 PM, Matt Corgan <[EMAIL PROTECTED]>
> wrote:
> >
> > > Did you try setting put.setWriteToWAL(false) as Bryan suggested?  This
> > may
> > > not be what you want in the end, but seeing what happens may help
> debug.
> > >
> > > Matt
> > >
> > > On Sat, Oct 13, 2012 at 8:58 AM, Jonathan Bishop <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Suraj,
> > > >
> > > > I bumped my regionservers all the way up to 32g from 8g. They are
> > running
> > > > on 64g and 128g machines on our cluster. Unfortunately, the machines
> > all
> > > > have various states of loading (usually high) from other users.
> > > >
> > > > In ganglia I do not see any swapping, but that has been known to
> happen
> > > > from time to time.
> > > >
> > > > Thanks for your help - I'll take a look at your links.
> > > >
> > > > Jon
> > > >
> > > > On Fri, Oct 12, 2012 at 7:30 PM, Suraj Varma <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Hi Jonathan:
> > > > > What specific metric on ganglia did you notice for "IO is spiking"?
> > Is
> > > > > it your disk IO? Is your disk swapping? Do you see cpu iowait