Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Generic increments?


+
Niels Basjes 2013-02-27, 13:47
Copy link to this message
-
Re: Generic increments?
We do not have such a facility as far as I know.
We have Increment/Append, and these work by locking the row, retrieving the old value, storing the updated value, unlocking the row.

-- Lars

________________________________
 From: Niels Basjes <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, February 27, 2013 5:47 AM
Subject: Generic increments?
 
Hi,

Last year at a meetup I spoke with Lars George about the counters in hbase.
What I understood is that the counters are stored as increments (i.e.
increment without locking) and during compaction and querying a the
increments are aggregated into the actual value.

So far I've examined the API and this seems to work as long as the value is
a long.

Now incrementing longs is nice but I would like to do things like
- Calculating min, max
- Bloomfilters
- Average ( recording both the "count" and "sum" )
- Variance and Standard Deviation ( using
http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm)

All of those need more bytes of internal storage and need custom code for
storing, aggregating and querying.
Especially querying because perhaps I can ask several different questions
to a single byte[].
If I store both the count and the sum in a single byte[] then I can ask
getN(), getSum(), getAvg()

Now my question to you guys is how I can implement such a more generic form
of "lock free increments" with user defined setters, getters and a custom
aggregator (used for both compacting and querying).
Perhaps there is an example on how to do this?

--
Best regards / Met vriendelijke groeten,

Niels Basjes
+
Niels Basjes 2013-02-28, 09:22
+
Nick Dimiduk 2013-03-04, 19:37
+
Niels Basjes 2013-03-04, 19:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB