Also wondering if the StumbleUpon folks are willing to merge it in as an alternative to HBase back end.
Eric Newton 2014-05-11, 21:04
It is being maintained. I have tried very hard not to modify the core OpenTSDB to support it. But, it would be nice if we could define a storage-independent layer to which it could adhere.
I don't believe the OTSDB team is interested, but a basic scalable back-end abstraction would be nice. As would a standard java build environment, but that doesn't seem to be wanted, either.
We could work towards a common storage abstraction, which would be a reasonable request.
Zipkin does a good job of being storage independent. I would work towards their model.
On May 11, 2014 10:28 AM, "Arshak Navruzyan" <[EMAIL PROTECTED]> wrote:
Donald Miner 2014-05-11, 23:29
Crazy/bad idea I've had before.... we could develop a hbase->accumulo proxy that receives basic hbase commands and then writes out accumulo.
David Medinets 2014-05-11, 23:34
Please define "hbase commands". On Sun, May 11, 2014 at 7:29 PM, Donald Miner <[EMAIL PROTECTED]>wrote:
Sean Busbey 2014-05-12, 00:18
I've started looking at something like this for the major read/write APIs for HBase and Accumulo in Kite.
One difference is that I'd prefer to eventually get to a common API that can be backed by either HBase 0.98+ or Accumulo.
Sean On May 11, 2014 6:34 PM, "David Medinets" <[EMAIL PROTECTED]> wrote:
Donald Miner 2014-05-12, 00:28
"Commands" like scan, put, create table, etc.
It would respond to hbase thrift and pretend it was hbase.... But it is using accumulo behind the scenes. basically be a translation layer. There are obviously some things that don't translate.
I presume you could do something across all of the big table implementations.
Mike Drob 2014-05-12, 00:31
The replication interface might be an easy place to try and support this. On Sun, May 11, 2014 at 8:20 PM, Donald Miner <[EMAIL PROTECTED]>wrote:
Sean Busbey 2014-05-12, 00:42
Yeah I was thinking covering most of the bigtable implementations at the source level would be an easy place to start.
Any interest in organizing a hack day near the Accumulo Summit to try to get an initial implementation done?
Sean On May 11, 2014 7:28 PM, "Donald Miner" <[EMAIL PROTECTED]> wrote:
Donald Miner 2014-05-12, 00:57
We are having a hackathon the day of during the conference. Not exclusive to the idea of having a separate hack day the day before. I know some of us are participating in the conference and can't participate in the hackathon.
Josh Elser 2014-05-12, 01:07
Possibly, but it would look different than Don described. It would be fairly easy to write data to an existing HBase instance from Accumulo (and one can probably assume the opposite to be true). Thus, Gets/Scans would be using the HBase API.
On 5/11/14, 8:30 PM, Mike Drob wrote:
Donald Miner 2014-05-12, 02:13
I've thought about this a lot actually (and finally have some time to sit down and type it out). This plays into my larger idea of an automated hbase->accumulo suite. A system that moves data from one to the other is one piece, the other piece is letting existing hbase applications hit accumulo instead. Anyways...
The most efficient thing to do is to have Accumulo tablet servers speak something that a native hbase client can talk to. What's going on in a client is pretty complex, so it's not going to be as easy as writing some interfaces. I think.
The proxy idea is okay, but I think we'll see some performance problems.
The other option is to do it client side, which would mean the applications would need to be rebuilt with our "special" APIs that look the same as before, but really behind the scenes are translating it to an accumulo call. Maybe they'd have to change an import or something.
Maybe there is a larger project here, something like a bigtable API that Cassandra, HBase, and Accumulo all can sign on to supporting. Or it is just client-side code. Not sure. Then something like OpenTSDB or zipkin would just write to the bigtable API and you wouldn't have to write new code.
Lot of options... not sure which one i like. Sorry for rambling.
-d On Sun, May 11, 2014 at 8:56 PM, Donald Miner <[EMAIL PROTECTED]>wrote: Donald Miner Chief Technology Officer ClearEdge IT Solutions, LLC Cell: 443 799 7807 www.clearedgeit.com