HBase, mail # user - Re: Bulk Loading from Oracle to Hbase

Re: Bulk Loading from Oracle to Hbase

Mehmet Simsek 2012-12-13, 15:57

Thanks to answer Mohammad.I tried sqoop but there are biliions rows. Select query processing too long.namenode does not get respond from daanode in 600sec. And throws exception. I can increase this duration but select query running too long may be 6 hours. Is there a another good solution? Or another technique in sqoop?

M.S

On 13 Ara 2012, at 17:48, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hello Mehmet,
>
> Have a look at Acache Sqoop <http://sqoop.apache.org/>
>
> Regards,
> Mohammad Tariq
>
>
>
> On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <[EMAIL PROTECTED]>wrote:
>
>> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
>> technique is the best for this bulk loading?
>>
>> Thanks
>>
>> M.S
>>
>>

Re: Bulk Loading from Oracle to Hbase

Manoj Babu 2012-12-13, 16:44

Mehmet,

You can try to write a MapReduce using DBInputFormat and insert into HBase.
http://hadoop.apache.org/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/db/DBInputFormat.html
Cheers!
Manoj.

On Thu, Dec 13, 2012 at 9:27 PM, Mehmet Simsek <[EMAIL PROTECTED]>wrote:

> Thanks to answer Mohammad.I tried sqoop but there are biliions rows.
> Select query processing too long.namenode does not get respond from daanode
> in 600sec. And throws exception. I can increase this duration but select
> query running too long may be 6 hours. Is there a another good solution? Or
> another technique in sqoop?
>
> M.S
>
> On 13 Ara 2012, at 17:48, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>
> > Hello Mehmet,
> >
> > Have a look at Acache Sqoop <http://sqoop.apache.org/>
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> >
> > On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <[EMAIL PROTECTED]
> >wrote:
> >
> >> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
> >> technique is the best for this bulk loading?
> >>
> >> Thanks
> >>
> >> M.S
> >>
> >>
>

Re: Bulk Loading from Oracle to Hbase

Amandeep Khurana 2012-12-13, 16:48

Mehmet

What's the problem you are getting while running the Sqoop job? Can
you give details?

-Amandeep

On Thu, Dec 13, 2012 at 8:44 AM, Manoj Babu <[EMAIL PROTECTED]> wrote:
> Mehmet,
>
> You can try to write a MapReduce using DBInputFormat and insert into HBase.
> http://hadoop.apache.org/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/db/DBInputFormat.html
>
>
> Cheers!
> Manoj.
>
>
>
> On Thu, Dec 13, 2012 at 9:27 PM, Mehmet Simsek <[EMAIL PROTECTED]>wrote:
>
>> Thanks to answer Mohammad.I tried sqoop but there are biliions rows.
>> Select query processing too long.namenode does not get respond from daanode
>> in 600sec. And throws exception. I can increase this duration but select
>> query running too long may be 6 hours. Is there a another good solution? Or
>> another technique in sqoop?
>>
>> M.S
>>
>> On 13 Ara 2012, at 17:48, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>
>> > Hello Mehmet,
>> >
>> > Have a look at Acache Sqoop <http://sqoop.apache.org/>
>> >
>> > Regards,
>> > Mohammad Tariq
>> >
>> >
>> >
>> > On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
>> >> technique is the best for this bulk loading?
>> >>
>> >> Thanks
>> >>
>> >> M.S
>> >>
>> >>
>>

Re: Bulk Loading from Oracle to Hbase

anil gupta 2012-12-13, 17:37

Hi Mehmet,

My suggestion would be:

1. Use Sqoop to copy the data from Oracle into HDFS.
2. Use HBase Bulk Loader to load the data into HBase Table.

Sqoop implementation of loading data directly into HBase uses HBase Api
which is not as efficient as HBase Bulk Loader. Also, features provided by
Sqoop for loading data into HBase are limited.

HTH,
Anil Gupta
On Thu, Dec 13, 2012 at 8:48 AM, Amandeep Khurana <[EMAIL PROTECTED]> wrote:

> Mehmet
>
> What's the problem you are getting while running the Sqoop job? Can
> you give details?
>
> -Amandeep
>
> On Thu, Dec 13, 2012 at 8:44 AM, Manoj Babu <[EMAIL PROTECTED]> wrote:
> > Mehmet,
> >
> > You can try to write a MapReduce using DBInputFormat and insert into
> HBase.
> >
> http://hadoop.apache.org/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/db/DBInputFormat.html
> >
> >
> > Cheers!
> > Manoj.
> >
> >
> >
> > On Thu, Dec 13, 2012 at 9:27 PM, Mehmet Simsek <[EMAIL PROTECTED]
> >wrote:
> >
> >> Thanks to answer Mohammad.I tried sqoop but there are biliions rows.
> >> Select query processing too long.namenode does not get respond from
> daanode
> >> in 600sec. And throws exception. I can increase this duration but select
> >> query running too long may be 6 hours. Is there a another good
> solution? Or
> >> another technique in sqoop?
> >>
> >> M.S
> >>
> >> On 13 Ara 2012, at 17:48, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> >>
> >> > Hello Mehmet,
> >> >
> >> > Have a look at Acache Sqoop <http://sqoop.apache.org/>
> >> >
> >> > Regards,
> >> > Mohammad Tariq
> >> >
> >> >
> >> >
> >> > On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <
> [EMAIL PROTECTED]
> >> >wrote:
> >> >
> >> >> Hi,I want to load 5 billion rows from oracle table to hbase table.
> Which
> >> >> technique is the best for this bulk loading?
> >> >>
> >> >> Thanks
> >> >>
> >> >> M.S
> >> >>
> >> >>
> >>
>

--
Thanks & Regards,
Anil Gupta

Re: Bulk Loading from Oracle to Hbase

Manoj Babu 2012-12-14, 03:35

can have look on this feature of hdfs.push from oracle R Connector.

http://docs.oracle.com/cd/E27101_01/doc.10/e27365/orhc.htm#CFHHEFHA

Cheers!
Manoj.

On Fri, Dec 14, 2012 at 8:41 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:

> Hello Mehmet,
>
> what about your oracle? a single node or RAC ?
>
>
> On Thu, Dec 13, 2012 at 11:46 PM, Mehmet Simsek <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,I want to load 5 billion rows from oracle table to hbase table. Which
> > technique is the best for this bulk loading?
> >
> > Thanks
> >
> > M.S
> >
> >
>

Re: Bulk Loading from Oracle to Hbase

Mehmet Simsek 2012-12-14, 03:36

Hello Azuryy,

Oracle in RAC.because of too long running sql statement datanode cannot respond to namenode about statistics.because of that exception is thrown by namenode like as "cannot respond from data node in 600sec..." In sqoop tool.

Is there an another solution about this problem?

M.S.

On 14 Ara 2012, at 05:11, Azuryy Yu <[EMAIL PROTECTED]> wrote:

> Hello Mehmet,
>
> what about your oracle? a single node or RAC ?
>
>
> On Thu, Dec 13, 2012 at 11:46 PM, Mehmet Simsek <[EMAIL PROTECTED]>wrote:
>
>> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
>> technique is the best for this bulk loading?
>>
>> Thanks
>>
>> M.S
>>
>>

Re: Bulk Loading from Oracle to Hbase

Azuryy Yu 2012-12-14, 05:50

Then, I suggest:
1) Increase your session per user in RAC equal or greater than your map
tasks
2) I suppose you want 100 map tasks, then write a little java code to
generate 100 files, each file has only one line, you can write any in this
line.
3) Define a STEP for all maps. then each map task select STEP records from
DB.
4) Put all your files to the HDFS
5) Create Map(no reduce here), execute "select * from Oracle where rownum
>= lineno and rownum < maptask_id * STEP" in each map task.
On Fri, Dec 14, 2012 at 11:36 AM, Mehmet Simsek <[EMAIL PROTECTED]>wrote:

> Hello Azuryy,
>
> Oracle in RAC.because of too long running sql statement datanode cannot
> respond to namenode about statistics.because of that exception is thrown by
> namenode like as "cannot respond from data node in 600sec..." In sqoop tool.
>
> Is there an another solution about this problem?
>
> M.S.
>
> On 14 Ara 2012, at 05:11, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>
> > Hello Mehmet,
> >
> > what about your oracle? a single node or RAC ?
> >
> >
> > On Thu, Dec 13, 2012 at 11:46 PM, Mehmet Simsek <
> [EMAIL PROTECTED]>wrote:
> >
> >> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
> >> technique is the best for this bulk loading?
> >>
> >> Thanks
> >>
> >> M.S
> >>
> >>
>