Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Multiple region servers per physical node


Copy link to this message
-
Re: Multiple region servers per physical node
Elliot,

  Can you elaborate on the blocking being a major factor?  Are you
referring to the default value of 7 slowing down writes?  I don't think
removing that feature is a great idea.  Here are a couple things that it is
helpful for:

1.) Slows down the write path so we are less likely to end up with 1000
storefiles per region server
2.) Slows down the write path enough for the end user to realize it is
slow, and thus start troubleshooting/optimizing the write path.

I think it might be fair to bump to 15 - 20 by default, but this is the
LAST option I touch when troubleshooting the write path.  Usually when we
start to have blocking issues it is caused by a few other issues:

1.) Too many regions for the allowed memstore upper/lower limit and we are
flushing too small
2.) Too small of Hlogs/number of HLogs and we are prematurely rolling
3.) Ingest rate is too fast for the memstore size and needs to be raised.
4.) Slow/not enough drives to keep up with the compaction churn and needs
to be tuned

The defaults for HBase out of the box here are decent for testing, but not
optimized for a production workload.  If this is not what you were talking
about sorry for the long rant :)
On Tue, Jul 30, 2013 at 2:34 AM, Elliott Clark <[EMAIL PROTECTED]> wrote:

> G1 doesn't really make our write path much better if you have uneven
> region writes (zipfian distribution or the like).
> Lately I've been seeing the memstore blocking size per region being a
> major factor.  In fact I'm thinking of opening a jira to remove it by
> default.
>
> On Mon, Jul 29, 2013 at 4:12 PM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
> > Having a difficult time avoiding evacuation failures under concurrent
> read
> > and write stress. Tuning helps but raises contention for CPU. This is
> with
> > "interesting" heap sizes - 32-128GB.
> >
> >
> > On Mon, Jul 29, 2013 at 3:44 PM, Bryan Beaudreault <
> [EMAIL PROTECTED]
> >> wrote:
> >
> >> Due to the contention for CPU resources?  Or why?
> >>
> >>
> >> On Mon, Jul 29, 2013 at 6:23 PM, Andrew Purtell <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > On Mon, Jul 29, 2013 at 3:14 PM, Elliott Clark <[EMAIL PROTECTED]>
> >> wrote:
> >> >
> >> > > Yes the G1 looks promising if you have a very read heavy workload
> >> > > (We've been running it for integration tests for about a month or
> so).
> >> > >
> >> >
> >> > That's my experience too. Unfortunately it's less promising once the
> >> > workload is mixed.
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> >    - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> > (via Tom White)
> >> >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>

--
Kevin O'Dell
Systems Engineer, Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB