Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> SILT - nice keyvalue store paper

Copy link to this message
RE: SILT - nice keyvalue store paper
Oh, and when running these experiments, you should look at the impact at which order they are run in, whether you run them multiple times per JVM instance, etc.  Basically, you need to be cognizant of the HotSpot optimizations the JVM is doing at runtime.

> -----Original Message-----
> From: Jonathan Gray [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, October 23, 2011 4:20 PM
> Subject: RE: SILT - nice keyvalue store paper
> Very nice experiment, Akash.  Keep getting your hands dirty and digging!  :)
> I think your results might change if you bump the test up to 1000 threads or
> so.  100 threads can still perform okay when there's a global lock but the
> contention at 1000 threads will kill you and that's when CSLM should do much
> better.  (1000 handler threads is approx. what I run with on RS in prod).
> Though I am a bit surprised that at 100 threads the TreeMap was significantly
> faster.  Your inconsistent results are a bit odd, you might try an order of
> magnitude more operations per thread.  You might also gather some
> statistics about tree size and per operation latency.
> I've done some isolated CSLM benchmarks in the past and have never been
> able to reproduce any of the slowness people suggest.  I recall trying some
> impractically large MemStores and everything still being quite fast.
> Over in Cassandra, I believe they have a two-level CSLM with the first map
> key being the row and then the columns for each row in their own CSLM.
> I've been told this is somewhat of a pain point for them.  And keep in mind
> they have one shard/region per node and we generally have several smaller
> MemStores on each node (tens to thousands).  Not sure we would want to
> try that.  There could be some interesting optimizations if you had very
> specific issues, like if you had a ton of reads to MemStore and not many
> writes you could keep some kind of mirrored hashmap.
> And for writes, the WAL is definitely the latency bottleneck.  But if you are
> doing lots of small operations, so your WALEdits are not large, and with some
> of the HLog batching features going in to trunk, you end up with hundreds of
> requests per HLog sync.  And although the syncs are higher latency, with
> batching you end up getting high throughput.  And the bottleneck shifts.
> Each sync will take approx. 1-5ms, so let's say 250 requests per HLog sync
> batch, 4ms per sync, so 62.5k req/sec.  (62.5k * 100 bytes/req = 600K/sec,
> very reasonable).  If you're mixing in reads as well (or if you're doing
> increments which do a read and write), then this adds to the CPU usage and
> contention without adding to HLog throughput.
> All of a sudden the bottleneck becomes CPU/contention and not HLog
> latency or throughput.  Highly concurrent increments/counters with a largely
> in-memory dataset can easily be CPU bottlenecked.
> For one specific application Dhruba and I worked on, we made some good
> improvements in CPU efficiency by reducing the number of operations and
> increasing efficiency on the CSLM.  Doing things like always taking a tailMap
> and working from that instead of starting at the root node, using an iterator()
> and taking advantage of the available remove() semantics, or simply just
> mutating things that are normally immutable :)  Unfortunately many of these
> optimizations were semi-horrid hacks and introduced things like
> ModifiableKeyValues, so they all haven't made their way to apache.
> In the end, after our optimizations, the real world workload Dhruba and I
> were working with was not all in-memory so the bottleneck in production
> became the random reads (so increasing the block cache hit ratio is the
> focus) rather than CPU contention or HLog throughput.
> JG
> From: Akash Ashok [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, October 23, 2011 2:57 AM
> Subject: Re: SILT - nice keyvalue store paper
> I was running some similar tests and came across a surprising finding. I