Hi everyone, I want to introduce our open-source project Themis which implements cross-row/corss-table transaction on HBase.
Themis follows google's percolator algorithm(http://research.google.com/pubs/pub36726.html), which provides ACID-compliant transaction and snapshot isolation. The cross-row transaction is based on HBase's single-row atomic semantics and doesn't use a central transaction server, so that supports linear-scalability.
Themis depends on a timestamp server to provides global strictly incremental timestamp to define the order of transactions, which will be used to resolve the write-write and read-write conflicts. The timestamp server is lightweight and could achieve hight throughput(500, 000 + qps), and Themis will batch timestamp requests across transactions in one Rpc, so that it won't become the bottleneck of the system even when processing billions of transactions every day.
Although Themis could be implemented totally in client-side, we adopt coprocessor framework of HBase to achieve higher performance. Themis includes a client-side library to provides transaction APIs, such as themisPut/themisGet/themisScan/themisDelete, and a coprocessor library loaded on regionserver. Therefore, Themis could be used without changing the code and logic of HBase.
We have been validating the correctness of Themis for a few months by a AccountTransfer simulation program, which concurrently does cross-row transactions by transferring money among different accounts(each account is a row in HBase) and verifies total money of all accounts doesn't change in the simulation. We have also run Themis on our production environment.
We test the performance of Themis and get comparable result as percolator. The single-column transaction represents the worst performance case for Themis compared with HBase, the result is: 1) For read, the performance of percolator is 90% of HBase; 2) For write, the performance of percolator is 23% of HBase. The write performance drops a lot because Themis uses two-phase commit protocol to achieve ACID of transaction. For multi-row write, we improve the performance by paralleling all writes of pre-write phase. For single-row write, we are optimizing two-phase commit protocol to achieve better performance and will update the result when it is ready. The details of performance result could be found in github.
This is great work, thank you for sharing. Please consider adding it to the Supported Projects section of the Apache HBase wiki: http://wiki.apache.org/hadoop/SupportingProjects On Tue, Jul 8, 2014 at 12:34 AM, 崔建伟 <[EMAIL PROTECTED]> wrote: Best regards,
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Themis : implements cross-row/corss-table transaction on HBase.