Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # user - can I bulk import multiple mysql tables to one HBase/Hive table


Copy link to this message
-
Re: can I bulk import multiple mysql tables to one HBase/Hive table
Venkat Ranganathan 2013-08-05, 21:43
Using the unions as Raghav says is the approach to take   I would suggest
doing the following

Create a query such as

    select name, id, address,... from (select user_name as name, user_id as
id, user_address as address ... union select customer_name as name,
customer_id as id,...union select employee_name as name,....) as tab
where \$CONDITIONS

and then use sqoop to do the query-import

You can also pass some conditions in the where clause if needed.   Please
see the sqoop documentation on making either Hive or HBase

Venkat

On Mon, Aug 5, 2013 at 2:06 PM, Raghav Gautam <[EMAIL PROTECTED]>wrote:

> Hi Shengjie,
>
> You can try importing using --query and --boundary-query options.
>
> Your query might look something like:
> select * from A UNION select * from B UNION select * from C
>
> And you boundary queries might look like:
> select min(mintemp) from (select min(id) as mintemp from A UNION select
> min(id) as mintemp from B UNION select min(id) as mintemp from C) t;
>
> select max(maxtemp) from (select max(id) as maxtemp from A UNION select
> max(id) as maxtemp from B UNION select max(id) as maxtemp from C) t;
>
> If possible you can try creating a view and import that.
>
> With Regards,
> Raghav.
>
>
>
> On Mon, Aug 5, 2013 at 1:02 AM, Shengjie Min <[EMAIL PROTECTED]> wrote:
>
>> Hi guys,
>>
>> I am reasonable new to Sqoop, just have a quick question here:
>>
>> If I have multiple similar mysql tables, e.g:
>>
>> table A: "users", columns: user_name, user_id, user_address, etc etc
>>
>> table B: "customers" columns: customer_name, customer_id,
>> customer_address, etc etc
>>
>> table C: "employee" columns: employee_name, employee_id,
>> employe_address, etc etc
>>
>> Is it possible that using Sqoop to import the three tables into one HBase
>> or Hive table? So After the import, I have one HBase table contains all the
>> records in table A, B, C ?
>>
>> --
>> All the best,
>> Shengjie Min
>>
>
>