Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Uneven write request to regions


Copy link to this message
-
Re: Uneven write request to regions
Thanks! Offline as in table disabled or cluster shutdown?

On Wednesday, November 20, 2013, Tom Brown wrote:

> The trade-off we make is to increase our write performance knowing it will
> negatively impact our read performance. In our case, however, we write a
> lot of rows that might never be read (depending on the specific deep-dive
> queries that will be run), so it's an ok trade-off. However, our layout is
> similar to the one described by Mike, so when we perform a read, we don't
> have to send to every region in the system, only the few that might have
> the data we need. Bloom filters also make random reads more efficient.
>
> Online merge is only available in the 0.96.x code, but an offline merge
> exists for 0.94.x (that may not be a viable option for you). From the
> command line:
>
> hbase org.apache.hadoop.hbase.util.Merge "table" "region1", "region2"
>
> However, if you have a specific weekly time that you can use for offline
> maintenance, writing a utility that splits the heavily used (hot) regions
> and merges the empty ones would allow you to balance your regions more
> appropriately across your cluster.
>
> --Tom
>
>
> On Wed, Nov 20, 2013 at 8:43 AM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
> > We use https://github.com/sematext/HBaseWD and I just learned
> > Amazon.com people are using it and are happy with it, so it may work
> > for you, too.
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Wed, Nov 20, 2013 at 1:00 AM, Asaf Mesika <[EMAIL PROTECTED]>
> > wrote:
> > > Thanks for clearing that out.
> > > I'm using your message to ping anyone who assist as to it appears the
> use
> > > case should happen to a lot of people?
> > >
> > > Thanks!
> > >
> > > On Wednesday, November 20, 2013, Himanshu Vashishtha wrote:
> > >
> > >> Re: "The 32 limit makes HBase go into
> > >> stress mode, and dump all involving regions contains in those 32 WAL
> > >> Files."
> > >>
> > >> Pardon, I haven't read all your data points/details thoroughly, but
> the
> > >> above statement is not true. Rather, it looks at the oldest WAL file,
> > and
> > >> flushes those regions which would free that WAL file.
> > >>
> > >> But I agree that in general with this kind of workload, we should
> handle
> > >> WAL files more intelligently and free up those WAL files which don't
> > have
> > >> any dependency (that is, all their entries are already flushed) when
> > >> archiving. We do that in trunk but not in any released version,
> though.
> > >>
> > >>
> > >>
> > >> On Sat, Nov 16, 2013 at 11:16 AM, Asaf Mesika <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > First I forgot to mention that <customerId> in our case is
> > >> > MD5(<customerId>).
> > >> > In our case, we have so much data flowing in, that we end up having
> a
> > >> > region per <customerId><bucket> pretty quickly and even that, is
> > splitted
> > >> > into different regions by specific date duration (timestamp).
> > >> >
> > >> > We're not witnessing a hotspot issue. I built some scripts in java
> and
> > >> awk,
> > >> > and saw that 66% of our customers use more than 1Rs.
> > >> >
> > >> > We have two main serious issues: primary and secondary.
> > >> >
> > >> > Our primary issue being the slow-region vs fast-region. First let's
> be
> > >> > reminded that a region represents as I detailed before a specific
> > >> > <customerId><bucket>. Some customers gets x50 times more data that
> > other
> > >> > customers at a specific time frame (2hrs - 1 day). So in a one RS,
> we
> > >> have
> > >> > regions getting 10 write requests per hour, vs 50k write requests
> per
> > >> hour.
> > >> > So the region mapped to the slow-filling customer id, doesn't get to
> > the
> > >> > 256MB flush limit and hence isn't flushed, while the regions mapped
> to
> > >> the
> > >> > fast-filling customer id, are flushing very quickly since they are
> > >> filling
>