Hi

I know HBase is designed for OLAP, query intensive type of applications. But
I like the flexibility feature of its column-base architecture which allows
me having no need to predefine every column of a table and I can dynamically
add new column with value in my OLTP application code and capture its meta
data information.

My question is basically about if we can use HBase for OLTP application
database. I know Hbase works well with Inserting column data of a row key
and set new version for the new piece of the data, and not so well for
updating and deleting existing piece of data. However, if I turn OLTP update
and delete operations into all insertion of new version of colum data as I
described below:
For OLTP data update, if I set my table column family’s versioning to 1 and
always do insert (put) when there is need to update an existing data row
columns, and let Hbase to handle the delete of the old versions through DB
garbage collection.
For OLTP data delete, I can use inserting new version on a flag field to
“deleted”, which is a logical delete, and have some batch job to clean up
all logically deleted rows later.

Will the above scenario work for using HBase for an OLTP application? Any
flaws on doing it?

Can some one share the experiences of using HBase for OLTP applications?

Thanks,

--
View this message in context: http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33107782.html
Sent from the HBase User mailing list archive at Nabble.com.

For starters, see the two video presentations on this page...

http://hbase.apache.org/book.html#other.info
On 1/9/12 12:25 PM, "fullysane" <[EMAIL PROTECTED]> wrote:

>
>Hi
>
>I know HBase is designed for OLAP, query intensive type of applications.
>But
>I like the flexibility feature of its column-base architecture which
>allows
>me having no need to predefine every column of a table and I can
>dynamically
>add new column with value in my OLTP application code and capture its meta
>data information.
>
>My question is basically about if we can use HBase for OLTP application
>database. I know Hbase works well with Inserting column data of a row key
>and set new version for the new piece of the data, and not so well for
>updating and deleting existing piece of data. However, if I turn OLTP
>update
>and delete operations into all insertion of new version of colum data as I
>described below:
>For OLTP data update, if I set my table column family¹s versioning to 1
>and
>always do insert (put) when there is need to update an existing data row
>columns, and let Hbase to handle the delete of the old versions through DB
>garbage collection.
>For OLTP data delete, I can use inserting new version on a flag field to
>³deleted², which is a logical delete, and have some batch job to clean up
>all logically deleted rows later.
>
>Will the above scenario work for using HBase for an OLTP application? Any
>flaws on doing it?
>
>Can some one share the experiences of using HBase for OLTP applications?
>
>Thanks,
>
>--
>View this message in context:
>http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33107782.ht
>ml
>Sent from the HBase User mailing list archive at Nabble.com.
>
>

Delete and Updates in HBase are like new writes.. The way to update a cell
is to actually do a Put. And when you delete, it internally flags the cell
to be deleted and removes the data from the underlying file on the next
compaction. If you want to learn the technical details further, you could
look at the source and also refer to the BigTable paper for the design
principles.

-Amandeep

On Tue, Jan 10, 2012 at 1:25 AM, fullysane <[EMAIL PROTECTED]> wrote:

>
> Hi
>
> I know HBase is designed for OLAP, query intensive type of applications.
> But
> I like the flexibility feature of its column-base architecture which allows
> me having no need to predefine every column of a table and I can
> dynamically
> add new column with value in my OLTP application code and capture its meta
> data information.
>
> My question is basically about if we can use HBase for OLTP application
> database. I know Hbase works well with Inserting column data of a row key
> and set new version for the new piece of the data, and not so well for
> updating and deleting existing piece of data. However, if I turn OLTP
> update
> and delete operations into all insertion of new version of colum data as I
> described below:
> For OLTP data update, if I set my table column family’s versioning to 1 and
> always do insert (put) when there is need to update an existing data row
> columns, and let Hbase to handle the delete of the old versions through DB
> garbage collection.
> For OLTP data delete, I can use inserting new version on a flag field to
> “deleted”, which is a logical delete, and have some batch job to clean up
> all logically deleted rows later.
>
> Will the above scenario work for using HBase for an OLTP application? Any
> flaws on doing it?
>
> Can some one share the experiences of using HBase for OLTP applications?
>
> Thanks,
>
> --
> View this message in context:
> http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33107782.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

And this...

http://hbase.apache.org/book.html#datamodel

On 1/9/12 12:36 PM, "Amandeep Khurana" <[EMAIL PROTECTED]> wrote:

>Delete and Updates in HBase are like new writes.. The way to update a cell
>is to actually do a Put. And when you delete, it internally flags the cell
>to be deleted and removes the data from the underlying file on the next
>compaction. If you want to learn the technical details further, you could
>look at the source and also refer to the BigTable paper for the design
>principles.
>
>-Amandeep
>
>On Tue, Jan 10, 2012 at 1:25 AM, fullysane <[EMAIL PROTECTED]> wrote:
>
>>
>> Hi
>>
>> I know HBase is designed for OLAP, query intensive type of applications.
>> But
>> I like the flexibility feature of its column-base architecture which
>>allows
>> me having no need to predefine every column of a table and I can
>> dynamically
>> add new column with value in my OLTP application code and capture its
>>meta
>> data information.
>>
>> My question is basically about if we can use HBase for OLTP application
>> database. I know Hbase works well with Inserting column data of a row
>>key
>> and set new version for the new piece of the data, and not so well for
>> updating and deleting existing piece of data. However, if I turn OLTP
>> update
>> and delete operations into all insertion of new version of colum data
>>as I
>> described below:
>> For OLTP data update, if I set my table column family¹s versioning to 1
>>and
>> always do insert (put) when there is need to update an existing data row
>> columns, and let Hbase to handle the delete of the old versions through
>>DB
>> garbage collection.
>> For OLTP data delete, I can use inserting new version on a flag field to
>> ³deleted², which is a logical delete, and have some batch job to clean
>>up
>> all logically deleted rows later.
>>
>> Will the above scenario work for using HBase for an OLTP application?
>>Any
>> flaws on doing it?
>>
>> Can some one share the experiences of using HBase for OLTP applications?
>>
>> Thanks,
>>
>> --
>> View this message in context:
>>
>>http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33107782.h
>>tml
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>

On Mon, Jan 9, 2012 at 9:25 AM, fullysane <[EMAIL PROTECTED]> wrote:
>
> Hi
>
> I know HBase is designed for OLAP, query intensive type of applications.

I would disagree. HBase isn't designed for OLAP at all - It's a way
better fit for the kind of applications you're referring to with
mostly single-row accesses.

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

> I know HBase is designed for OLAP, query intensive type of applications.

That is not entirely true. HBase is a pure transaction system and does OLTP
workloads for us. We probably more than 2 millions ops/sec for one of our
application, details here:
https://www.facebook.com/note.php?note_id=454991608919

-dhruba
On Mon, Jan 9, 2012 at 9:25 AM, fullysane <[EMAIL PROTECTED]> wrote:

>
> Hi
>
> I know HBase is designed for OLAP, query intensive type of applications.
> But
> I like the flexibility feature of its column-base architecture which allows
> me having no need to predefine every column of a table and I can
> dynamically
> add new column with value in my OLTP application code and capture its meta
> data information.
>
> My question is basically about if we can use HBase for OLTP application
> database. I know Hbase works well with Inserting column data of a row key
> and set new version for the new piece of the data, and not so well for
> updating and deleting existing piece of data. However, if I turn OLTP
> update
> and delete operations into all insertion of new version of colum data as I
> described below:
> For OLTP data update, if I set my table column family’s versioning to 1 and
> always do insert (put) when there is need to update an existing data row
> columns, and let Hbase to handle the delete of the old versions through DB
> garbage collection.
> For OLTP data delete, I can use inserting new version on a flag field to
> “deleted”, which is a logical delete, and have some batch job to clean up
> all logically deleted rows later.
>
> Will the above scenario work for using HBase for an OLTP application? Any
> flaws on doing it?
>
> Can some one share the experiences of using HBase for OLTP applications?
>
> Thanks,
>
> --
> View this message in context:
> http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33107782.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
--
Subscribe to my posts at http://www.facebook.com/dhruba

All,

Just my $0.02 worth of 'expertise'...

1) Just because you can do something doesn't mean you should.
2) One should always try to use the right tool for the job regardless of your 'fashion sense'.
3) Just because someone says "Facebook or Yahoo! does X", doesn't mean its a good idea, or its the right choice for you and your team.

Having said that...

Yes, you can use HBase to handle OLTP queries. However you do not have transactional capabilities built in such that you will have to manage them within your application.
Not really an easy task when you think about it.  It really depends on what you want to do with your OLTP system. Hotel reservation systems not really a good idea....

There are some inherent problems with HBase in an OLTP environment.

1) Eventual consistency. You can google the CAP theorem and you'll see why this is an issue.
2) Lack of transaction support. Note: Row Level Locking that is in HBase has nothing to do with Row Level Locking with respect to transactional support.  
3) HBase size and scale vs RDBMS. For OLTP, RDBMS is the best tool for the job. So why do you want to use HBase over what one could call the 'defacto' standard?
The point here on #3 is that the normal tool of choice is an RDBMS. So you really, really need to justify why you're not going with this. I mean there could be a valid reason, but in most cases no.

Where dhruba indicates that HBase is a pure transaction system, and does support OLTP workloads... absolutely Not!

So what I suggest is that if you want to do OLTP in HBase, the first thing you have to do is to prove that you can't solve the problem in an RDBMs.

Having said all that... I'm going to shut now... ;-)

-Mike

> Date: Mon, 9 Jan 2012 10:55:45 -0800
> Subject: Re: Question about HBase for OLTP
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
> CC: [EMAIL PROTECTED]
>
> > I know HBase is designed for OLAP, query intensive type of applications.
>
> That is not entirely true. HBase is a pure transaction system and does OLTP
> workloads for us. We probably more than 2 millions ops/sec for one of our
> application, details here:
> https://www.facebook.com/note.php?note_id=454991608919
>
> -dhruba
>
>
> On Mon, Jan 9, 2012 at 9:25 AM, fullysane <[EMAIL PROTECTED]> wrote:
>
> >
> > Hi
> >
> > I know HBase is designed for OLAP, query intensive type of applications.
> > But
> > I like the flexibility feature of its column-base architecture which allows
> > me having no need to predefine every column of a table and I can
> > dynamically
> > add new column with value in my OLTP application code and capture its meta
> > data information.
> >
> > My question is basically about if we can use HBase for OLTP application
> > database. I know Hbase works well with Inserting column data of a row key
> > and set new version for the new piece of the data, and not so well for
> > updating and deleting existing piece of data. However, if I turn OLTP
> > update
> > and delete operations into all insertion of new version of colum data as I
> > described below:
> > For OLTP data update, if I set my table column family’s versioning to 1 and
> > always do insert (put) when there is need to update an existing data row
> > columns, and let Hbase to handle the delete of the old versions through DB
> > garbage collection.
> > For OLTP data delete, I can use inserting new version on a flag field to
> > “deleted”, which is a logical delete, and have some batch job to clean up
> > all logically deleted rows later.
> >
> > Will the above scenario work for using HBase for an OLTP application? Any
> > flaws on doing it?
> >
> > Can some one share the experiences of using HBase for OLTP applications?
> >
> > Thanks,
> >
> > --
> > View this message in context:
> > http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33107782.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
> >
>
>
> --
> Subscribe to my posts at http://www.facebook.com/dhruba
     

Uhmmm. Well...  It depends on your data and what you want to do...

Can you fit all of the data into a single row?
Does it make sense to use a sequence file for the raw data and then use HBase to maintain indexes?

Just some food for thought.

> From: [EMAIL PROTECTED]
> Date: Mon, 9 Jan 2012 10:19:16 -0800
> Subject: Re: Question about HBase for OLTP
> To: [EMAIL PROTECTED]
>
> On Mon, Jan 9, 2012 at 9:25 AM, fullysane <[EMAIL PROTECTED]> wrote:
> >
> > Hi
> >
> > I know HBase is designed for OLAP, query intensive type of applications.
>
> I would disagree. HBase isn't designed for OLAP at all - It's a way
> better fit for the kind of applications you're referring to with
> mostly single-row accesses.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
     

1) Eventual Consistency isn't a problem here.  HBase is a strict
consistency system.  Maybe you have us confused with other Dynamo-based
Open Source projects?
2) MySQL and other traditional RDBMS systems are definitely a lot more
solid, well-tested, and subtlety tuned than HBase.  The vast majority (if
not all) of database systems developed in the past decade have this
provlem.  HBase has 2 main advantages over a traditional RDBMS workload
for OLTP:
  A. Large-scale workloads : Facebook Messages have a constant growing set
of data that is 1PB+.  And we're growing at 250MB/month.  This is hard to
manage this with a traditional RDBMS.  Logical database sharding is
extremely useful.
  B. Write-dominated workloads : Examples like time-series databases, user
analytics, etc are very write-heavy. A LSMT approach is architecturally
better than a B-tree approach.  Having done system testing internally, we
already see IOPS advantage with HBase over MySQL in writes.
3) A big question is what you need out of a database system.  Most web
companies are worried about the 'large-scale workloads' problem if their
site becomes popular, so a working familiarity with a distributed database
system for less mission-critical applications is worthwhile even if the
performance and reliability isn't there yet.
4) If you have any mission-critical data, you really should think about a
disaster recovery plan outside of HBase, which is not as critical with a
traditional RDBMS.  Facebook Messages ends up using Scribe as a backup
mechanism.  We are currently working on HBase Snapshots to allow disaster
recovery with HBase alone, but you shouldn't hedge bets on it being
completed within your timeframe.
On 1/9/12 2:31 PM, "Michael Segel" <[EMAIL PROTECTED]> wrote:

>
>
>All,
>
>Just my $0.02 worth of 'expertise'...
>
>1) Just because you can do something doesn't mean you should.
>2) One should always try to use the right tool for the job regardless of
>your 'fashion sense'.
>3) Just because someone says "Facebook or Yahoo! does X", doesn't mean
>its a good idea, or its the right choice for you and your team.
>
>Having said that...
>
>Yes, you can use HBase to handle OLTP queries. However you do not have
>transactional capabilities built in such that you will have to manage
>them within your application.
>Not really an easy task when you think about it.  It really depends on
>what you want to do with your OLTP system. Hotel reservation systems not
>really a good idea....
>
>There are some inherent problems with HBase in an OLTP environment.
>
>1) Eventual consistency. You can google the CAP theorem and you'll see
>why this is an issue.
>2) Lack of transaction support. Note: Row Level Locking that is in HBase
>has nothing to do with Row Level Locking with respect to transactional
>support.  
>3) HBase size and scale vs RDBMS. For OLTP, RDBMS is the best tool for
>the job. So why do you want to use HBase over what one could call the
>'defacto' standard?
>The point here on #3 is that the normal tool of choice is an RDBMS. So
>you really, really need to justify why you're not going with this. I mean
>there could be a valid reason, but in most cases no.
>
>Where dhruba indicates that HBase is a pure transaction system, and does
>support OLTP workloads... absolutely Not!
>
>So what I suggest is that if you want to do OLTP in HBase, the first
>thing you have to do is to prove that you can't solve the problem in an
>RDBMs.
>
>Having said all that... I'm going to shut now... ;-)
>
>-Mike
>
>
>
>> Date: Mon, 9 Jan 2012 10:55:45 -0800
>> Subject: Re: Question about HBase for OLTP
>> From: [EMAIL PROTECTED]
>> To: [EMAIL PROTECTED]
>> CC: [EMAIL PROTECTED]
>>
>> > I know HBase is designed for OLAP, query intensive type of
>>applications.
>>
>> That is not entirely true. HBase is a pure transaction system and does
>>OLTP
>> workloads for us. We probably more than 2 millions ops/sec for one of
>>our
>> application, details here:
>> https://www.facebook.com/note.php?note_id=454991608919

Ok..

Look, here's the thing... HBase has no transactional support.
OLTP systems like PoS systems, Hotel Reservation Systems, Trading systems... among others really need this.

Again, I can't stress this point enough... DO NOT THINK ABOUT USING HBASE AS AN OLTP SYSTEM UNLESS YOU HAVE ALREADY GONE THROUGH THE PROCESS OF DEMONSTRATING WHY YOU CAN NOT DO THIS IN AN RDBMS.
Here Nicolas proves my point. FB in their example says that they couldn't fit their messages in to an RDBMs.

Since HBase lacks transactional support you need to deal with security so that only your application is touching those tables which are part of your OLTP system.

Again, I cannot stress the importance in understanding that the term ROW LEVEL LOCKING means one thing to HBase developers and another to RDBMs types.

So I'll shut up now and let you young whipper snappers walk in to the mine field.  ;-)
The obvious take what I say with a grain of Kosher Salt and YMMV are included....

Enjoy

-Mike
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
> Subject: Re: Question about HBase for OLTP
> Date: Mon, 9 Jan 2012 23:18:39 +0000
>
>
> 1) Eventual Consistency isn't a problem here.  HBase is a strict
> consistency system.  Maybe you have us confused with other Dynamo-based
> Open Source projects?
> 2) MySQL and other traditional RDBMS systems are definitely a lot more
> solid, well-tested, and subtlety tuned than HBase.  The vast majority (if
> not all) of database systems developed in the past decade have this
> provlem.  HBase has 2 main advantages over a traditional RDBMS workload
> for OLTP:
>   A. Large-scale workloads : Facebook Messages have a constant growing set
> of data that is 1PB+.  And we're growing at 250MB/month.  This is hard to
> manage this with a traditional RDBMS.  Logical database sharding is
> extremely useful.
>   B. Write-dominated workloads : Examples like time-series databases, user
> analytics, etc are very write-heavy. A LSMT approach is architecturally
> better than a B-tree approach.  Having done system testing internally, we
> already see IOPS advantage with HBase over MySQL in writes.
> 3) A big question is what you need out of a database system.  Most web
> companies are worried about the 'large-scale workloads' problem if their
> site becomes popular, so a working familiarity with a distributed database
> system for less mission-critical applications is worthwhile even if the
> performance and reliability isn't there yet.
> 4) If you have any mission-critical data, you really should think about a
> disaster recovery plan outside of HBase, which is not as critical with a
> traditional RDBMS.  Facebook Messages ends up using Scribe as a backup
> mechanism.  We are currently working on HBase Snapshots to allow disaster
> recovery with HBase alone, but you shouldn't hedge bets on it being
> completed within your timeframe.
>
>
> On 1/9/12 2:31 PM, "Michael Segel" <[EMAIL PROTECTED]> wrote:
>
> >
> >
> >All,
> >
> >Just my $0.02 worth of 'expertise'...
> >
> >1) Just because you can do something doesn't mean you should.
> >2) One should always try to use the right tool for the job regardless of
> >your 'fashion sense'.
> >3) Just because someone says "Facebook or Yahoo! does X", doesn't mean
> >its a good idea, or its the right choice for you and your team.
> >
> >Having said that...
> >
> >Yes, you can use HBase to handle OLTP queries. However you do not have
> >transactional capabilities built in such that you will have to manage
> >them within your application.
> >Not really an easy task when you think about it.  It really depends on
> >what you want to do with your OLTP system. Hotel reservation systems not
> >really a good idea....
> >
> >There are some inherent problems with HBase in an OLTP environment.
> >
> >1) Eventual consistency. You can google the CAP theorem and you'll see
> >why this is an issue.
> >2) Lack of transaction support. Note: Row Level Locking that is in HBase
> >has nothing to do with Row Level Locking with respect to transactional
     

I think that Amandeep pretty much nailed the intent of the original
question with his response "Delete and Updates in HBase are like new
writes.." since I think one of the central questions was about over-write
behavior (also covered in DataModel section), and the subsequent delete
isn't required because with versions=1 it will get cleaned out with major
compaction.

As for being conservative with HBase, I agree with that and for the record
that's pointed out here...
http://hbase.apache.org/book.html#arch.overview

... with the section "When Should I Use HBase."  I'll add something about
(lack of) transactional support in there as well.
On 1/9/12 7:37 PM, "Michael Segel" <[EMAIL PROTECTED]> wrote:

>
>Ok..
>
>Look, here's the thing... HBase has no transactional support.
>OLTP systems like PoS systems, Hotel Reservation Systems, Trading
>systems... among others really need this.
>
>Again, I can't stress this point enough... DO NOT THINK ABOUT USING HBASE
>AS AN OLTP SYSTEM UNLESS YOU HAVE ALREADY GONE THROUGH THE PROCESS OF
>DEMONSTRATING WHY YOU CAN NOT DO THIS IN AN RDBMS.
>Here Nicolas proves my point. FB in their example says that they couldn't
>fit their messages in to an RDBMs.
>
>Since HBase lacks transactional support you need to deal with security so
>that only your application is touching those tables which are part of
>your OLTP system.
>
>Again, I cannot stress the importance in understanding that the term ROW
>LEVEL LOCKING means one thing to HBase developers and another to RDBMs
>types.
>
>So I'll shut up now and let you young whipper snappers walk in to the
>mine field.  ;-)
>The obvious take what I say with a grain of Kosher Salt and YMMV are
>included....
>
>Enjoy
>
>-Mike
>
>
>> From: [EMAIL PROTECTED]
>> To: [EMAIL PROTECTED]
>> Subject: Re: Question about HBase for OLTP
>> Date: Mon, 9 Jan 2012 23:18:39 +0000
>>
>>
>> 1) Eventual Consistency isn't a problem here.  HBase is a strict
>> consistency system.  Maybe you have us confused with other Dynamo-based
>> Open Source projects?
>> 2) MySQL and other traditional RDBMS systems are definitely a lot more
>> solid, well-tested, and subtlety tuned than HBase.  The vast majority
>>(if
>> not all) of database systems developed in the past decade have this
>> provlem.  HBase has 2 main advantages over a traditional RDBMS workload
>> for OLTP:
>>   A. Large-scale workloads : Facebook Messages have a constant growing
>>set
>> of data that is 1PB+.  And we're growing at 250MB/month.  This is hard
>>to
>> manage this with a traditional RDBMS.  Logical database sharding is
>> extremely useful.
>>   B. Write-dominated workloads : Examples like time-series databases,
>>user
>> analytics, etc are very write-heavy. A LSMT approach is architecturally
>> better than a B-tree approach.  Having done system testing internally,
>>we
>> already see IOPS advantage with HBase over MySQL in writes.
>> 3) A big question is what you need out of a database system.  Most web
>> companies are worried about the 'large-scale workloads' problem if their
>> site becomes popular, so a working familiarity with a distributed
>>database
>> system for less mission-critical applications is worthwhile even if the
>> performance and reliability isn't there yet.
>> 4) If you have any mission-critical data, you really should think about
>>a
>> disaster recovery plan outside of HBase, which is not as critical with a
>> traditional RDBMS.  Facebook Messages ends up using Scribe as a backup
>> mechanism.  We are currently working on HBase Snapshots to allow
>>disaster
>> recovery with HBase alone, but you shouldn't hedge bets on it being
>> completed within your timeframe.
>>
>>
>> On 1/9/12 2:31 PM, "Michael Segel" <[EMAIL PROTECTED]> wrote:
>>
>> >
>> >
>> >All,
>> >
>> >Just my $0.02 worth of 'expertise'...
>> >
>> >1) Just because you can do something doesn't mean you should.
>> >2) One should always try to use the right tool for the job regardless
>>of
>> >your 'fashion sense'.

Yeah, look, here's the hard part.

I don't want to be a debbie downer, or someone who constantly says... its not a good idea.

I really want to encourage people to think about what they are doing, and conceptually how HBase is going to handle that process.
The more you think, the better your design will be.

I also believe in HDD or DDD (Hangover Driven Design) and (Drunk Driven Design). Its no mystery why these 'hackathons' where massive quantities of pizza and beer are so successful at creating something. :-)

If you don't need transactions or locks...  then there's a lot of cool things you can do.
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
> Subject: Re: Question about HBase for OLTP
> Date: Tue, 10 Jan 2012 01:34:13 +0000
>
>
> I think that Amandeep pretty much nailed the intent of the original
> question with his response "Delete and Updates in HBase are like new
> writes.." since I think one of the central questions was about over-write
> behavior (also covered in DataModel section), and the subsequent delete
> isn't required because with versions=1 it will get cleaned out with major
> compaction.
>
> As for being conservative with HBase, I agree with that and for the record
> that's pointed out here...
>
>
> http://hbase.apache.org/book.html#arch.overview
>
> ... with the section "When Should I Use HBase."  I'll add something about
> (lack of) transactional support in there as well.
>
>
>
>
> On 1/9/12 7:37 PM, "Michael Segel" <[EMAIL PROTECTED]> wrote:
>
> >
> >Ok..
> >
> >Look, here's the thing... HBase has no transactional support.
> >OLTP systems like PoS systems, Hotel Reservation Systems, Trading
> >systems... among others really need this.
> >
> >Again, I can't stress this point enough... DO NOT THINK ABOUT USING HBASE
> >AS AN OLTP SYSTEM UNLESS YOU HAVE ALREADY GONE THROUGH THE PROCESS OF
> >DEMONSTRATING WHY YOU CAN NOT DO THIS IN AN RDBMS.
> >Here Nicolas proves my point. FB in their example says that they couldn't
> >fit their messages in to an RDBMs.
> >
> >Since HBase lacks transactional support you need to deal with security so
> >that only your application is touching those tables which are part of
> >your OLTP system.
> >
> >Again, I cannot stress the importance in understanding that the term ROW
> >LEVEL LOCKING means one thing to HBase developers and another to RDBMs
> >types.
> >
> >So I'll shut up now and let you young whipper snappers walk in to the
> >mine field.  ;-)
> >The obvious take what I say with a grain of Kosher Salt and YMMV are
> >included....
> >
> >Enjoy
> >
> >-Mike
> >
> >
> >> From: [EMAIL PROTECTED]
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: Question about HBase for OLTP
> >> Date: Mon, 9 Jan 2012 23:18:39 +0000
> >>
> >>
> >> 1) Eventual Consistency isn't a problem here.  HBase is a strict
> >> consistency system.  Maybe you have us confused with other Dynamo-based
> >> Open Source projects?
> >> 2) MySQL and other traditional RDBMS systems are definitely a lot more
> >> solid, well-tested, and subtlety tuned than HBase.  The vast majority
> >>(if
> >> not all) of database systems developed in the past decade have this
> >> provlem.  HBase has 2 main advantages over a traditional RDBMS workload
> >> for OLTP:
> >>   A. Large-scale workloads : Facebook Messages have a constant growing
> >>set
> >> of data that is 1PB+.  And we're growing at 250MB/month.  This is hard
> >>to
> >> manage this with a traditional RDBMS.  Logical database sharding is
> >> extremely useful.
> >>   B. Write-dominated workloads : Examples like time-series databases,
> >>user
> >> analytics, etc are very write-heavy. A LSMT approach is architecturally
> >> better than a B-tree approach.  Having done system testing internally,
> >>we
> >> already see IOPS advantage with HBase over MySQL in writes.
> >> 3) A big question is what you need out of a database system.  Most web
> >> companies are worried about the 'large-scale workloads' problem if their
     

On Mon, Jan 9, 2012 at 4:37 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>
> Ok..
>
> Look, here's the thing... HBase has no transactional support.
> OLTP systems like PoS systems, Hotel Reservation Systems, Trading systems... among others really need this.
>
> Again, I can't stress this point enough... DO NOT THINK ABOUT USING HBASE AS AN OLTP SYSTEM UNLESS YOU HAVE ALREADY GONE THROUGH THE PROCESS OF DEMONSTRATING WHY YOU CAN NOT DO THIS IN AN RDBMS.
> Here Nicolas proves my point. FB in their example says that they couldn't fit their messages in to an RDBMs.

I agree with your points, but he wrote that it would be hard, not that
it could not be done.
https://www.facebook.com/MySQLatFacebook

--
Mark Callaghan
[EMAIL PROTECTED]

Hi dhruba:

Thank you for the information. Can you let me know the distribution of
insert, update, delete, and query of your OLTP transactions? Are they mostly
Insert and query? how do you set up/configure the Hbase table's column
family (versioning, compression, ...) for your OLTP application?

Thanks,

Fuesane
Dhruba Borthakur-2 wrote:
>
>> I know HBase is designed for OLAP, query intensive type of applications.
>
> That is not entirely true. HBase is a pure transaction system and does
> OLTP
> workloads for us. We probably more than 2 millions ops/sec for one of our
> application, details here:
> https://www.facebook.com/note.php?note_id=454991608919
>
> -dhruba
>
>
> On Mon, Jan 9, 2012 at 9:25 AM, fullysane <[EMAIL PROTECTED]> wrote:
>
>>
>> Hi
>>
>> I know HBase is designed for OLAP, query intensive type of applications.
>> But
>> I like the flexibility feature of its column-base architecture which
>> allows
>> me having no need to predefine every column of a table and I can
>> dynamically
>> add new column with value in my OLTP application code and capture its
>> meta
>> data information.
>>
>> My question is basically about if we can use HBase for OLTP application
>> database. I know Hbase works well with Inserting column data of a row key
>> and set new version for the new piece of the data, and not so well for
>> updating and deleting existing piece of data. However, if I turn OLTP
>> update
>> and delete operations into all insertion of new version of colum data as
>> I
>> described below:
>> For OLTP data update, if I set my table column family’s versioning to 1
>> and
>> always do insert (put) when there is need to update an existing data row
>> columns, and let Hbase to handle the delete of the old versions through
>> DB
>> garbage collection.
>> For OLTP data delete, I can use inserting new version on a flag field to
>> “deleted”, which is a logical delete, and have some batch job to clean up
>> all logically deleted rows later.
>>
>> Will the above scenario work for using HBase for an OLTP application? Any
>> flaws on doing it?
>>
>> Can some one share the experiences of using HBase for OLTP applications?
>>
>> Thanks,
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33107782.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Subscribe to my posts at http://www.facebook.com/dhruba
>
>

--
View this message in context: http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33128144.html
Sent from the HBase User mailing list archive at Nabble.com.

Hi Mike:

Thereason I am thinking Hbase for OLTP is that I need a column-based
(key-value pair)OLTP DBMS which allows me not to predefine columns for a
table and can add new column to a table on the fly like Hbase does. Alougth
this can be done in any RDBMS with so called skinny and tall table contain
only key and value columns and then pivoting the data to look like a
traditionla spead sheet like table, but the pivoting perforamnce will be a
drag for query.

Any idea to solve this problem?

Thanks,
fullysane wrote:
>
> Hi
>
> I know HBase is designed for OLAP, query intensive type of applications.
> But I like the flexibility feature of its column-base architecture which
> allows me having no need to predefine every column of a table and I can
> dynamically add new column with value in my OLTP application code and
> capture its meta data information.
>
> My question is basically about if we can use HBase for OLTP application
> database. I know Hbase works well with Inserting column data of a row key
> and set new version for the new piece of the data, and not so well for
> updating and deleting existing piece of data. However, if I turn OLTP
> update and delete operations into all insertion of new version of colum
> data as I described below:
> For OLTP data update, if I set my table column family’s versioning to 1
> and always do insert (put) when there is need to update an existing data
> row columns, and let Hbase to handle the delete of the old versions
> through DB garbage collection.
> For OLTP data delete, I can use inserting new version on a flag field to
> “deleted”, which is a logical delete, and have some batch job to clean up
> all logically deleted rows later.
>
> Will the above scenario work for using HBase for an OLTP application? Any
> flaws on doing it?
>
> Can some one share the experiences of using HBase for OLTP applications?
>
> Thanks,
>
>

--
View this message in context: http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33128226.html
Sent from the HBase User mailing list archive at Nabble.com.

Here are some of our stats of FB messages on HBase:

6B+ messages/day

Traffic to HBase
75+ Billion  R+W ops/day
At peak: 1.5M ops/sec
~ 55% Read vs. 45% Write ops

Avg write op inserts ~16 records across multiple column families and column
family updates in the same record are atomic (inside the same transaction)

-dhruba
On Thu, Jan 12, 2012 at 6:50 AM, fullysane <[EMAIL PROTECTED]> wrote:

>
> Hi Mike:
>
> Thereason I am thinking Hbase for OLTP is that I need a column-based
> (key-value pair)OLTP DBMS which allows me not to predefine columns for a
> table and can add new column to a table on the fly like Hbase does. Alougth
> this can be done in any RDBMS with so called skinny and tall table contain
> only key and value columns and then pivoting the data to look like a
> traditionla spead sheet like table, but the pivoting perforamnce will be a
> drag for query.
>
> Any idea to solve this problem?
>
> Thanks,
>
>
>
>
> fullysane wrote:
> >
> > Hi
> >
> > I know HBase is designed for OLAP, query intensive type of applications.
> > But I like the flexibility feature of its column-base architecture which
> > allows me having no need to predefine every column of a table and I can
> > dynamically add new column with value in my OLTP application code and
> > capture its meta data information.
> >
> > My question is basically about if we can use HBase for OLTP application
> > database. I know Hbase works well with Inserting column data of a row key
> > and set new version for the new piece of the data, and not so well for
> > updating and deleting existing piece of data. However, if I turn OLTP
> > update and delete operations into all insertion of new version of colum
> > data as I described below:
> > For OLTP data update, if I set my table column family’s versioning to 1
> > and always do insert (put) when there is need to update an existing data
> > row columns, and let Hbase to handle the delete of the old versions
> > through DB garbage collection.
> > For OLTP data delete, I can use inserting new version on a flag field to
> > “deleted”, which is a logical delete, and have some batch job to clean up
> > all logically deleted rows later.
> >
> > Will the above scenario work for using HBase for an OLTP application? Any
> > flaws on doing it?
> >
> > Can some one share the experiences of using HBase for OLTP applications?
> >
> > Thanks,
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33128226.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
--
Subscribe to my posts at http://www.facebook.com/dhruba

Mark,

There are a lot of things that have gone in to RDBMs where it was the only tool available, although not the best tool. You can put unstructured data in to some RDBMs. Its a pain but it can be done.

The point I am trying to make, which has also been made by myself and others on numerous threads...

It doesn't make sense to use HBase for a problem that could be solved easier in an RDBMs. Its actually counter productive.

What I am suggesting is that you study the problem and determine the best set of *available* tools to do the job.
> Date: Thu, 12 Jan 2012 06:22:03 -0800
> Subject: Re: Question about HBase for OLTP
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> On Mon, Jan 9, 2012 at 4:37 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
> >
> > Ok..
> >
> > Look, here's the thing... HBase has no transactional support.
> > OLTP systems like PoS systems, Hotel Reservation Systems, Trading systems... among others really need this.
> >
> > Again, I can't stress this point enough... DO NOT THINK ABOUT USING HBASE AS AN OLTP SYSTEM UNLESS YOU HAVE ALREADY GONE THROUGH THE PROCESS OF DEMONSTRATING WHY YOU CAN NOT DO THIS IN AN RDBMS.
> > Here Nicolas proves my point. FB in their example says that they couldn't fit their messages in to an RDBMs.
>
> I agree with your points, but he wrote that it would be hard, not that
> it could not be done.
> https://www.facebook.com/MySQLatFacebook
>
> --
> Mark Callaghan
> [EMAIL PROTECTED]
     

Dhruba:

Thank you for the stats.

Fuesane

Dhruba Borthakur-2 wrote:
>
> Here are some of our stats of FB messages on HBase:
>
> 6B+ messages/day
>
> Traffic to HBase
> 75+ Billion  R+W ops/day
> At peak: 1.5M ops/sec
> ~ 55% Read vs. 45% Write ops
>
> Avg write op inserts ~16 records across multiple column families and
> column
> family updates in the same record are atomic (inside the same transaction)
>
> -dhruba
>
>
>
>
> On Thu, Jan 12, 2012 at 6:50 AM, fullysane <[EMAIL PROTECTED]> wrote:
>
>>
>> Hi Mike:
>>
>> Thereason I am thinking Hbase for OLTP is that I need a column-based
>> (key-value pair)OLTP DBMS which allows me not to predefine columns for a
>> table and can add new column to a table on the fly like Hbase does.
>> Alougth
>> this can be done in any RDBMS with so called skinny and tall table
>> contain
>> only key and value columns and then pivoting the data to look like a
>> traditionla spead sheet like table, but the pivoting perforamnce will be
>> a
>> drag for query.
>>
>> Any idea to solve this problem?
>>
>> Thanks,
>>
>>
>>
>>
>> fullysane wrote:
>> >
>> > Hi
>> >
>> > I know HBase is designed for OLAP, query intensive type of
>> applications.
>> > But I like the flexibility feature of its column-base architecture
>> which
>> > allows me having no need to predefine every column of a table and I can
>> > dynamically add new column with value in my OLTP application code and
>> > capture its meta data information.
>> >
>> > My question is basically about if we can use HBase for OLTP application
>> > database. I know Hbase works well with Inserting column data of a row
>> key
>> > and set new version for the new piece of the data, and not so well for
>> > updating and deleting existing piece of data. However, if I turn OLTP
>> > update and delete operations into all insertion of new version of colum
>> > data as I described below:
>> > For OLTP data update, if I set my table column family’s versioning to 1
>> > and always do insert (put) when there is need to update an existing
>> data
>> > row columns, and let Hbase to handle the delete of the old versions
>> > through DB garbage collection.
>> > For OLTP data delete, I can use inserting new version on a flag field
>> to
>> > “deleted”, which is a logical delete, and have some batch job to clean
>> up
>> > all logically deleted rows later.
>> >
>> > Will the above scenario work for using HBase for an OLTP application?
>> Any
>> > flaws on doing it?
>> >
>> > Can some one share the experiences of using HBase for OLTP
>> applications?
>> >
>> > Thanks,
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33128226.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Subscribe to my posts at http://www.facebook.com/dhruba
>
>

--
View this message in context: http://old.nabble.com/Question-about-HBase-for-OLTP-tp33107782p33136155.html
Sent from the HBase User mailing list archive at Nabble.com.