Dear All,

We are going to do our experiment of a scientific papers, ]
We must insert data in our database for later consideration, it almost
300 tables each one has 2/000/000 records.
as you know It takes lots of time to do it with a single machine,
we are going to use our Hadoop cluster (32 machines) and divide 300
insertion tasks between them,
I need some hint to progress faster,
1- as i know we dont need to Reduser, just Mapper in enough.
2- so wee need just implement Mapper class with needed code.

Please let me know if there is any point,

Best Regards
Masoud

Dear All,

We are going to do our experiment of a scientific papers, ]
We must insert data in our database for later consideration, it almost
300 tables each one has 2/000/000 records.
as you know It takes lots of time to do it with a single machine,
we are going to use our Hadoop cluster (32 machines) and divide 300
insertion tasks between them,
I need some hint to progress faster,
1- as i know we dont need to Reduser, just Mapper in enough.
2- so wee need just implement Mapper class with needed code.

Please let me know if there is any point,

Best Regards
Masoud

Hello Masoud,

          You can use the Bulk Load feature. You might find it more
efficient than normal client APIs or using the TableOutputFormat.

The bulk load feature uses a MapReduce job to output table data
in HBase's internal data format, and then directly loads the
generated StoreFiles into a running cluster. Using bulk load will use
less CPU and network resources than simply using the HBase API.

For a detailed info you can go here :
http://hbase.apache.org/book/arch.bulk.load.html

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Mon, Feb 18, 2013 at 5:00 PM, Masoud <[EMAIL PROTECTED]> wrote:

>
> Dear All,
>
> We are going to do our experiment of a scientific papers, ]
> We must insert data in our database for later consideration, it almost
> 300 tables each one has 2/000/000 records.
> as you know It takes lots of time to do it with a single machine,
> we are going to use our Hadoop cluster (32 machines) and divide 300
> insertion tasks between them,
> I need some hint to progress faster,
> 1- as i know we dont need to Reduser, just Mapper in enough.
> 2- so wee need just implement Mapper class with needed code.
>
> Please let me know if there is any point,
>
> Best Regards
> Masoud
>
>
>
>

What database is this ? Was hbase mentioned ?

On Monday, February 18, 2013, Mohammad Tariq wrote:

> Hello Masoud,
>
>           You can use the Bulk Load feature. You might find it more
> efficient than normal client APIs or using the TableOutputFormat.
>
> The bulk load feature uses a MapReduce job to output table data
> in HBase's internal data format, and then directly loads the
> generated StoreFiles into a running cluster. Using bulk load will use
> less CPU and network resources than simply using the HBase API.
>
> For a detailed info you can go here :
> http://hbase.apache.org/book/arch.bulk.load.html
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Mon, Feb 18, 2013 at 5:00 PM, Masoud <[EMAIL PROTECTED]<javascript:_e({}, 'cvml', '[EMAIL PROTECTED]');>
> > wrote:
>
>>
>> Dear All,
>>
>> We are going to do our experiment of a scientific papers, ]
>> We must insert data in our database for later consideration, it almost
>> 300 tables each one has 2/000/000 records.
>> as you know It takes lots of time to do it with a single machine,
>> we are going to use our Hadoop cluster (32 machines) and divide 300
>> insertion tasks between them,
>> I need some hint to progress faster,
>> 1- as i know we dont need to Reduser, just Mapper in enough.
>> 2- so wee need just implement Mapper class with needed code.
>>
>> Please let me know if there is any point,
>>
>> Best Regards
>> Masoud
>>
>>
>>
>>
>

Nope HBase wasn't mentioned.
The OP could be talking about using external tables and Hive.

The OP could still be stuck in the RDBMs world and hasn't flattened his data yet.
2 million records? Kinda small dontcha think?
Not Enough Information ...

On Feb 18, 2013, at 8:58 AM, Hemanth Yamijala <[EMAIL PROTECTED]> wrote:

> What database is this ? Was hbase mentioned ?
>
> On Monday, February 18, 2013, Mohammad Tariq wrote:
> Hello Masoud,
>  
>           You can use the Bulk Load feature. You might find it more
> efficient than normal client APIs or using the TableOutputFormat.
>
> The bulk load feature uses a MapReduce job to output table data
> in HBase's internal data format, and then directly loads the
> generated StoreFiles into a running cluster. Using bulk load will use
> less CPU and network resources than simply using the HBase API.
>
> For a detailed info you can go here :
> http://hbase.apache.org/book/arch.bulk.load.html
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Mon, Feb 18, 2013 at 5:00 PM, Masoud <[EMAIL PROTECTED]> wrote:
>
> Dear All,
>
> We are going to do our experiment of a scientific papers, ]
> We must insert data in our database for later consideration, it almost
> 300 tables each one has 2/000/000 records.
> as you know It takes lots of time to do it with a single machine,
> we are going to use our Hadoop cluster (32 machines) and divide 300
> insertion tasks between them,
> I need some hint to progress faster,
> 1- as i know we dont need to Reduser, just Mapper in enough.
> 2- so wee need just implement Mapper class with needed code.
>
> Please let me know if there is any point,
>
> Best Regards
> Masoud
>
>
>
>

Hello Tariq,

Our database is sql server 2008,
and we dont need to develop a professional app, we just need to develop
it fast and make our experiment result soon.
Thanks
On 02/18/2013 11:58 PM, Hemanth Yamijala wrote:
> What database is this ? Was hbase mentioned ?
>
> On Monday, February 18, 2013, Mohammad Tariq wrote:
>
>     Hello Masoud,
>               You can use the Bulk Load feature. You might find it more
>     efficient than normal client APIs or using the TableOutputFormat.
>
>     The bulk load feature uses a MapReduce job to output table data
>     in HBase's internal data format, and then directly loads the
>     generated StoreFiles into a running cluster. Using bulk load will use
>     less CPU and network resources than simply using the HBase API.
>
>     For a detailed info you can go here :
>     http://hbase.apache.org/book/arch.bulk.load.html
>
>     Warm Regards,
>     Tariq
>     https://mtariq.jux.com/
>     cloudfront.blogspot.com <http://cloudfront.blogspot.com>
>
>
>     On Mon, Feb 18, 2013 at 5:00 PM, Masoud
>     <[EMAIL PROTECTED] <javascript:_e({}, 'cvml',
>     '[EMAIL PROTECTED]');>> wrote:
>
>
>         Dear All,
>
>         We are going to do our experiment of a scientific papers, ]
>         We must insert data in our database for later consideration,
>         it almost
>         300 tables each one has 2/000/000 records.
>         as you know It takes lots of time to do it with a single machine,
>         we are going to use our Hadoop cluster (32 machines) and
>         divide 300
>         insertion tasks between them,
>         I need some hint to progress faster,
>         1- as i know we dont need to Reduser, just Mapper in enough.
>         2- so wee need just implement Mapper class with needed code.
>
>         Please let me know if there is any point,
>
>         Best Regards
>         Masoud
>
>
>
>
--
Masoud Reyhani Hamedani
Ph.D. Candidate
Department of Electronics and Computer Engineering, Hanyang University
Data Mining and Knowledge Engineering Lab,
Room 803 IT/BT Building 17
Haengdang-dong, Sungdong-gu Seoul, Republic of Korea, 133-791
Tel: +82-2-2220-4567
[EMAIL PROTECTED]
http://agape.hanyang.ac.kr

Hello Masoud,

Did you look at sqoop (http://sqoop.apache.org)? Maybe it can help you.

I think there also is a specific FileInputFormat designed for databases. I don't remember the name. If you use it, you need just write a mapper.

Guillaume Polaert | Cyrès Conseil
-----Message d'origine-----
De : Masoud [mailto:[EMAIL PROTECTED]]
Envoyé : lundi 18 février 2013 12:20
À : [EMAIL PROTECTED]
Objet : Database insertion by HAdoop

Dear All,

We are going to do our experiment of a scientific papers, ] We must insert data in our database for later consideration, it almost
300 tables each one has 2/000/000 records.
as you know It takes lots of time to do it with a single machine, we are going to use our Hadoop cluster (32 machines) and divide 300 insertion tasks between them, I need some hint to progress faster,
1- as i know we dont need to Reduser, just Mapper in enough.
2- so wee need just implement Mapper class with needed code.

Please let me know if there is any point,

Best Regards
Masoud