Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HTable writeAsyncBuffer


Copy link to this message
-
Re: HTable writeAsyncBuffer
It was written to be generic, but limited to 'put' to maintain the backward
compatibility.
Some 'Row' do not implement 'heapSize', so we have a limitation for some
types for the moment (we need the objects to implement heapSize as we need
to know how when it's time to flush the buffer). This can be solved type by
type for the ones that don't implement 'heapSize'
Server side, I'm not sure that all types are supported. it's not possible
to batch "Increment" for example. This can be solved type by type as well.

For the deletes case, I would expect that adding only deletes to the buffer
would work out of the box. It you mix puts and delete, I don't know. It's
worth trying I would say (and add a unit test in the code code if it does).
Adding delete would mean adding an option such as "autoFlushDelete", and
then using the existing buffer. It would make sense to add this
functionality to HBase imho. And this would be backward compatible as well.
You're likely to know it already, but just in case: you can already batch
deletes, and the batch mode uses the AsyncProcess code.

So:
> are there downsides if we were to add all these operations to the
writeAsyncBuffer?
> are there any usages of HTable.put that rely on it being send off instead
of being put in a buffer?
It's a user decision. If it can be controlled by a flag per HTable instance
such as 'autoFlush' today, it's fine. We can't extend autoFlush to all
operations as it would break backward compatibility, but adding flag per
operation type is fine imho.

Cheers,

Nicolas
On Tue, Dec 10, 2013 at 12:16 AM, Stack <[EMAIL PROTECTED]> wrote:

> On Mon, Dec 9, 2013 at 2:57 PM, Koert Kuipers <[EMAIL PROTECTED]> wrote:
>
> > are there downsides if we were to add all these operations to the
> > writeAsyncBuffer? are there any usages of HTable.put that rely on it
> being
> > send off instead of being put in a buffer?
> >
> >
> You can ask for a flush after adding a put if you have such a case (and
> there is autoflush flag).
> St.Ack
>