Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> gauging cost of region movement


Copy link to this message
-
RE: gauging cost of region movement
Also, using more stable measures of request count will help, such as 30 minute rolling averages.

> -----Original Message-----
> From: Jonathan Gray [mailto:[EMAIL PROTECTED]]
> Sent: Monday, March 21, 2011 4:23 PM
> To: [EMAIL PROTECTED]
> Subject: RE: gauging cost of region movement
>
> This is an interesting direction, and definitely file a JIRA as this could be an
> additional metric in the future, but it's not exactly what I had in mind.
>
> One of the hardest parts of load balancing based on request count and other
> dynamic/transient measures is that you can get some pretty pathological
> conditions where you are always moving stuff around.
>
> To guard against it, I think we'll need to move to more of a cost-based
> algorithm that is taking not just the difference in request counts into account
> but also a baseline "cost" of moving a region.  The cost difference in load
> between two unbalanced servers would have to outweigh the cost
> associated with moving a region.  As you say, looking at the number of live
> operations to a given region could contribute to the cost of moving that
> region, but the best measure for that is probably just looking at request
> count (it's all requests that incur a cost, not just active scanners).
>
> JG
>
> > -----Original Message-----
> > From: Ted Yu [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, March 21, 2011 3:44 PM
> > To: [EMAIL PROTECTED]
> > Subject: gauging cost of region movement
> >
> > Can we add a counter for the number of InternalScanner's to HRegion ?
> > We decrement this counter when close() is called.
> >
> > Such counter can be used to gauge the cost of moving the underlying
> region.
> >
> > Cheers
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB