Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # dev >> Many locality groups


+
Josh Elser 2013-09-18, 15:02
+
Keith Turner 2013-09-18, 15:35
Copy link to this message
-
Re: Many locality groups
Neat!

Glad to see I wasn't completely off base with some of the complexity
numbers I was expecting. I'll pick up my poking and prodding where you left
off.

Thanks, Keith.
On Wed, Sep 18, 2013 at 11:35 AM, Keith Turner <[EMAIL PROTECTED]> wrote:

> I ran some test before and after partitioning tablet memory in
> ACCUMULO-112.  I commented on the performance numbers I saw.  I checked in
> the code I used to test.
>
> test/src/main/java/org/apache/accumulo/test/IMMLGBenchmark.java
>
> Looking back at the test, one thing I did not time was reading all of the
> locality groups in scan.
>
>
> On Wed, Sep 18, 2013 at 11:02 AM, Josh Elser <[EMAIL PROTECTED]> wrote:
>
> > I have a use case in which I'm investigating setting a locality group on
> > every column family in a table which has very "dense" rows (many columns
> > appear within the same tablet).
> >
> > When scanning over a single column, I see a slow-down as one might expect
> > (filtering out the columns I don't care about). Setting each column into
> > its own locality group helps speed things up again for that single column
> > query case.
> >
> > I'm curious if anyone has any insight to when/if I'm going to start
> paying
> > a penalty for having many locality groups. Glancing back over
> RFile.Reader,
> > I have to read each LocalityGroupMetadata and its multi-level index
> (which
> > shouldn't be bad if I remember Keith's talks) and then I should get
> log(n)
> > reads across the locality groups I need to open.
> >
> > Is the same true for writing data to many a table with many locality
> > groups? Nothing terrible pops out at me looking at the code.
> >
> > I was planning to write some tests to try and simulate this, but figured
> I
> > can poll the community as well to see if anyone has experimented in this
> > scenario before.
> >
> > Thanks!
> >
> > - Josh
> >
>
+
Josh Elser 2013-09-18, 19:41
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB