Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Uneven write request to regions


Copy link to this message
-
Re: Uneven write request to regions
Thanks! Offline as in table disabled or cluster shutdown?

On Wednesday, November 20, 2013, Tom Brown wrote:

> The trade-off we make is to increase our write performance knowing it will
> negatively impact our read performance. In our case, however, we write a
> lot of rows that might never be read (depending on the specific deep-dive
> queries that will be run), so it's an ok trade-off. However, our layout is
> similar to the one described by Mike, so when we perform a read, we don't
> have to send to every region in the system, only the few that might have
> the data we need. Bloom filters also make random reads more efficient.
>
> Online merge is only available in the 0.96.x code, but an offline merge
> exists for 0.94.x (that may not be a viable option for you). From the
> command line:
>
> hbase org.apache.hadoop.hbase.util.Merge "table" "region1", "region2"
>
> However, if you have a specific weekly time that you can use for offline
> maintenance, writing a utility that splits the heavily used (hot) regions
> and merges the empty ones would allow you to balance your regions more
> appropriately across your cluster.
>
> --Tom
>
>
> On Wed, Nov 20, 2013 at 8:43 AM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
> > We use https://github.com/sematext/HBaseWD and I just learned
> > Amazon.com people are using it and are happy with it, so it may work
> > for you, too.
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Wed, Nov 20, 2013 at 1:00 AM, Asaf Mesika <[EMAIL PROTECTED]>
> > wrote:
> > > Thanks for clearing that out.
> > > I'm using your message to ping anyone who assist as to it appears the
> use
> > > case should happen to a lot of people?
> > >
> > > Thanks!
> > >
> > > On Wednesday, November 20, 2013, Himanshu Vashishtha wrote:
> > >
> > >> Re: "The 32 limit makes HBase go into
> > >> stress mode, and dump all involving regions contains in those 32 WAL
> > >> Files."
> > >>
> > >> Pardon, I haven't read all your data points/details thoroughly, but
> the
> > >> above statement is not true. Rather, it looks at the oldest WAL file,
> > and
> > >> flushes those regions which would free that WAL file.
> > >>
> > >> But I agree that in general with this kind of workload, we should
> handle
> > >> WAL files more intelligently and free up those WAL files which don't
> > have
> > >> any dependency (that is, all their entries are already flushed) when
> > >> archiving. We do that in trunk but not in any released version,
> though.
> > >>
> > >>
> > >>
> > >> On Sat, Nov 16, 2013 at 11:16 AM, Asaf Mesika <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > First I forgot to mention that <customerId> in our case is
> > >> > MD5(<customerId>).
> > >> > In our case, we have so much data flowing in, that we end up having
> a
> > >> > region per <customerId><bucket> pretty quickly and even that, is
> > splitted
> > >> > into different regions by specific date duration (timestamp).
> > >> >
> > >> > We're not witnessing a hotspot issue. I built some scripts in java
> and
> > >> awk,
> > >> > and saw that 66% of our customers use more than 1Rs.
> > >> >
> > >> > We have two main serious issues: primary and secondary.
> > >> >
> > >> > Our primary issue being the slow-region vs fast-region. First let's
> be
> > >> > reminded that a region represents as I detailed before a specific
> > >> > <customerId><bucket>. Some customers gets x50 times more data that
> > other
> > >> > customers at a specific time frame (2hrs - 1 day). So in a one RS,
> we
> > >> have
> > >> > regions getting 10 write requests per hour, vs 50k write requests
> per
> > >> hour.
> > >> > So the region mapped to the slow-filling customer id, doesn't get to
> > the
> > >> > 256MB flush limit and hence isn't flushed, while the regions mapped
> to
> > >> the
> > >> > fast-filling customer id, are flushing very quickly since they are
> > >> filling
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB