I have around 20 GB of data to be dumped into a hbase table.

Initially, I had a simple java program to put the values in a batch of
(5000-10000) records.  I tried concurrent inserts and each insert took about
15 seconds to write.  Which is very slow and was taking ages.

Next approach was to use importtsv, this started off with a set of maps and
after few minutes, I started getting RetriesException and errors out in a
while.

Of these experiments, I noticed that the master node was handing all the
traffic.  I understand that initially it dumps data in one node and then
splits across multiple nodes as data comes in.  Is there a way to split this
across regions in the beginning?

Or any other thoughts on how to handle inserts of large amounts of data?
Viv

Have you tried out the mix of importtsv + completebulkload? Would that
work for you?

On Sat, Mar 19, 2011 at 9:18 PM, Vivek Krishna <[EMAIL PROTECTED]> wrote:
> I have around 20 GB of data to be dumped into a hbase table.
>
> Initially, I had a simple java program to put the values in a batch of
> (5000-10000) records.  I tried concurrent inserts and each insert took about
> 15 seconds to write.  Which is very slow and was taking ages.
>
> Next approach was to use importtsv, this started off with a set of maps and
> after few minutes, I started getting RetriesException and errors out in a
> while.
>
> Of these experiments, I noticed that the master node was handing all the
> traffic.  I understand that initially it dumps data in one node and then
> splits across multiple nodes as data comes in.  Is there a way to split this
> across regions in the beginning?
>
> Or any other thoughts on how to handle inserts of large amounts of data?
> Viv
>

--
Harsh J
http://harshj.com

> Is there a way to split this
> across regions in the beginning?

Since you didn't mention your HBase version, I'm assuming you are
using 0.90.1 or later.
If so, yes, there is a way to pre-split the regions. See this:
http://hbase.apache.org/book/important_configurations.html#d0e1975

Also - as Harsh mentioned, the bulkload tool might be even better, so
take a look at that as well:
http://hbase.apache.org/bulk-loads.html

--Suraj

On Sat, Mar 19, 2011 at 8:48 AM, Vivek Krishna <[EMAIL PROTECTED]> wrote:
> I have around 20 GB of data to be dumped into a hbase table.
>
> Initially, I had a simple java program to put the values in a batch of
> (5000-10000) records.  I tried concurrent inserts and each insert took about
> 15 seconds to write.  Which is very slow and was taking ages.
>
> Next approach was to use importtsv, this started off with a set of maps and
> after few minutes, I started getting RetriesException and errors out in a
> while.
>
> Of these experiments, I noticed that the master node was handing all the
> traffic.  I understand that initially it dumps data in one node and then
> splits across multiple nodes as data comes in.  Is there a way to split this
> across regions in the beginning?
>
> Or any other thoughts on how to handle inserts of large amounts of data?
> Viv
>

Thanks Harsh..

Viv

On Sat, Mar 19, 2011 at 11:52 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Have you tried out the mix of importtsv + completebulkload? Would that
> work for you?
>
> On Sat, Mar 19, 2011 at 9:18 PM, Vivek Krishna <[EMAIL PROTECTED]>
> wrote:
> > I have around 20 GB of data to be dumped into a hbase table.
> >
> > Initially, I had a simple java program to put the values in a batch of
> > (5000-10000) records.  I tried concurrent inserts and each insert took
> about
> > 15 seconds to write.  Which is very slow and was taking ages.
> >
> > Next approach was to use importtsv, this started off with a set of maps
> and
> > after few minutes, I started getting RetriesException and errors out in a
> > while.
> >
> > Of these experiments, I noticed that the master node was handing all the
> > traffic.  I understand that initially it dumps data in one node and then
> > splits across multiple nodes as data comes in.  Is there a way to split
> this
> > across regions in the beginning?
> >
> > Or any other thoughts on how to handle inserts of large amounts of data?
> > Viv
> >
>
>
>
> --
> Harsh J
> http://harshj.com
>

Hi,

I want to do bulk loads by following http://hbase.apache.org/bulk-loads.html to create HFiles, and then using LoadIncrementalHFiles to load the data into a table.  Suppose the data I'm loading is being written to a new column that hasn't been used, and the rows are a superset of the rows already in the table.  Is it correct to assume that the existing data will not be affected by the load and that reads and writes can be happening during the load?  In other words, is the bulk load conceptually the same as doing a bunch of puts all at once through the api, or do I need to think of it differently?

Thanks,
Albert