Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> HBase - Secondary Index


Copy link to this message
-
Re: HBase - Secondary Index
I would like to reply to this...As previously i was part of the impl of
this..Hope Anoop corrects me if i am going wrong here..
.
 However if the update to the index table (after a succesful update of the
source table) fails for some other reason (without a crash of the region
server), the HLog will not be replayed..

As you understood the WALEdit is done for the index region and also for the
main region using WAL hooks.  The next step includes the addition to the
memstore..
So the KVs needs to be added to the memstore of both the main region and
index region. What type of failure do you foresee here?
You think of flushes or something that could fail?

If you see the Put() api code once the WAL edit is successfull there is no
need to rollback also. Just after the memstore addition happens for the
main table new hooks were added to make an entry for the index table.
 Ideally here both should pass. Also some addtional work was done inorder
to take into account the MVCC part.  So that a flush of the main region or
index region does not affect an incoming put or vice versa.

Anyway Anoop can answer to this more specifically as i dont have access to
the src code anymore.

Regards
Ram
On Wed, Dec 5, 2012 at 8:42 PM, Jan Van Besien <[EMAIL PROTECTED]> wrote:

> On 12/05/2012 12:04 PM, Anoop Sam John wrote:
>
>>           Yes we guarentee the consistency between user table and index
>> table. The put operation will be handled as a transactional way so as to
>> make sure the data is added to both tables or reverted back from both. Some
>> new CP hooks we have added for this obviously.
>>
>
> Would you be interested in sharing how exactly you guarantee this
> consistency? What are the CP hooks that you added and how exactly are they
> used?
>
> I can currently only guess how it could work in your implementation. For
> examply it could be that an update is a single WALEdit, which results in an
> update to both the source and index table. If the region server crashes
> between the update to the source and the index table, the HLog will be
> replied and thus you will have a chance to recover. However if the update
> to the index table (after a succesful update of the source table) fails for
> some other reason (without a crash of the region server), the HLog will not
> be replayed..
>

>
> Anyway, the above is just one assumption of your implementation could
> work. If you could share more details of the actual implementation, this
> would be helpful.
>
> Thanks
> Jan
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB