Hello,

Let me introduce new effort around HBase: HBaseHUT.
It suggests solution to mentioned many times on this mailing list problem
"do Get on every Put operation to update record" (which causes bad write
performance) and suitable for many use-cases.

Sources available here: http://github.com/sematext/HBaseHUT
Wiki with some idea/usage details: https://github.com/sematext/HBaseHUT/wiki

It would be great to receive the feedback on the overall idea.

Thank you!

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase

Thanks for sending announcement to the list Alex.  Congrats on new
project.  You might consider adding your project to
http://wiki.apache.org/hadoop/SupportingProjects?

St.Ack

On Tue, Nov 30, 2010 at 5:40 AM, Alex Baranau <[EMAIL PROTECTED]> wrote:
> Hello,
>
> Let me introduce new effort around HBase: HBaseHUT.
> It suggests solution to mentioned many times on this mailing list problem
> "do Get on every Put operation to update record" (which causes bad write
> performance) and suitable for many use-cases.
>
> Sources available here: http://github.com/sematext/HBaseHUT
> Wiki with some idea/usage details: https://github.com/sematext/HBaseHUT/wiki
>
> It would be great to receive the feedback on the overall idea.
>
> Thank you!
>
> Alex Baranau
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
>

Is there any limitation?

Many thanks

Alvin
On 1 December 2010 01:39, Stack <[EMAIL PROTECTED]> wrote:

> Thanks for sending announcement to the list Alex.  Congrats on new
> project.  You might consider adding your project to
> http://wiki.apache.org/hadoop/SupportingProjects?
>
> St.Ack
>
> On Tue, Nov 30, 2010 at 5:40 AM, Alex Baranau <[EMAIL PROTECTED]>
> wrote:
> > Hello,
> >
> > Let me introduce new effort around HBase: HBaseHUT.
> > It suggests solution to mentioned many times on this mailing list problem
> > "do Get on every Put operation to update record" (which causes bad write
> > performance) and suitable for many use-cases.
> >
> > Sources available here: http://github.com/sematext/HBaseHUT
> > Wiki with some idea/usage details:
> https://github.com/sematext/HBaseHUT/wiki
> >
> > It would be great to receive the feedback on the overall idea.
> >
> > Thank you!
> >
> > Alex Baranau
> > ----
> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
> HBase
> >
>

Hi Alvin,

There's no limitations, but in particular cases you won't want to use
HBaseHUT.

0) In general, if Get & Put operation for each write of new data is OK for
you (e.g. low write rate) then it's easier to go that way I believe.

1) In certain cases read performance can degrade noticeably when using
HBaseHUT (in the benefit of much higher write speed of course). By "certain
cases" I primarily mean the situation when both true of:
* updates affect just small amount of records (i.e. not far from
"well-spread") *and*
* updates (not "new inserts") rate is very heavy

Thus, if you have 100K of update operations (non-processed, i.e. updates
processing haven't fired for them yet) for some data item, then reading this
data item is likely to be slow (depending on your update logic, close to the
HBase scan speed on your cluster).
Note, that with "storing processed updates on client reads" feature next
time user requests for the same data item it will be returned very fast (as
single Get operation).

Thanks for the question, I'll gather them for the FAQ ;).

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase

On Wed, Dec 1, 2010 at 4:54 AM, Alvin C.L Huang <[EMAIL PROTECTED]>wrote:

> Is there any limitation?
>
> Many thanks
>
> Alvin
>
>
> On 1 December 2010 01:39, Stack <[EMAIL PROTECTED]> wrote:
>
> > Thanks for sending announcement to the list Alex.  Congrats on new
> > project.  You might consider adding your project to
> > http://wiki.apache.org/hadoop/SupportingProjects?
> >
> > St.Ack
> >
> > On Tue, Nov 30, 2010 at 5:40 AM, Alex Baranau <[EMAIL PROTECTED]>
> > wrote:
> > > Hello,
> > >
> > > Let me introduce new effort around HBase: HBaseHUT.
> > > It suggests solution to mentioned many times on this mailing list
> problem
> > > "do Get on every Put operation to update record" (which causes bad
> write
> > > performance) and suitable for many use-cases.
> > >
> > > Sources available here: http://github.com/sematext/HBaseHUT
> > > Wiki with some idea/usage details:
> > https://github.com/sematext/HBaseHUT/wiki
> > >
> > > It would be great to receive the feedback on the overall idea.
> > >
> > > Thank you!
> > >
> > > Alex Baranau
> > > ----
> > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
> > HBase
> > >
> >
>

 Hi,

could you describe the logic behind your implementation. I haven't found
any related documents on your wiki.

cheers
--
Renaud Delbru

On 30/11/10 13:40, Alex Baranau wrote:
> Hello,
>
> Let me introduce new effort around HBase: HBaseHUT.
> It suggests solution to mentioned many times on this mailing list problem
> "do Get on every Put operation to update record" (which causes bad write
> performance) and suitable for many use-cases.
>
> Sources available here: http://github.com/sematext/HBaseHUT
> Wiki with some idea/usage details: https://github.com/sematext/HBaseHUT/wiki
>
> It would be great to receive the feedback on the overall idea.
>
> Thank you!
>
> Alex Baranau
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
>

Well, the implementation is quite simple (for now): there are few classes in
sources. You can take a look and provide a feedback (would be great)!

Briefly (and simplified), the idea is the following:

Writing new data:
When new data is arrived (in form of a Put) it is stored as a separate
record in HBase (magic postfix is added to the key so that it doesn't
override existing record with the same key which is meant to be updated).

Reading data:
When user fetches data using scan pending updates (if there are any) are
processed. Thus user always gets latest data.
User can also perform processing on e.g. some periodic basis to prevent too
much work during data scans. Currently user can process updates for
particular records or set of records (the set is determined by simple Scan).

Updates Processing internals:
When updates processing is fired, the records with the same original keys
are passed to UpdateProcessor (UP) implementation (defined by user) as
iterable list. UP outputs updated record which is stored back in HBase (if
this was requested) with updated postfix and processed records are scheduled
for remove. New postfix tells what data has been processed to preserve
reading consistency (as writing updated record and deleting processed is not
atomic).

Hope this gives some picture. I'd encourage you to use
https://groups.google.com/group/hbasehut to discuss details. Thank you!

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase

On Wed, Dec 1, 2010 at 6:51 PM, Renaud Delbru <[EMAIL PROTECTED]>wrote:

>  Hi,
>
> could you describe the logic behind your implementation. I haven't found
> any related documents on your wiki.
>
> cheers
> --
> Renaud Delbru
>
>
> On 30/11/10 13:40, Alex Baranau wrote:
>
>> Hello,
>>
>> Let me introduce new effort around HBase: HBaseHUT.
>> It suggests solution to mentioned many times on this mailing list problem
>> "do Get on every Put operation to update record" (which causes bad write
>> performance) and suitable for many use-cases.
>>
>> Sources available here: http://github.com/sematext/HBaseHUT
>> Wiki with some idea/usage details:
>> https://github.com/sematext/HBaseHUT/wiki
>>
>> It would be great to receive the feedback on the overall idea.
>>
>> Thank you!
>>
>> Alex Baranau
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
>> HBase
>>
>>
>