Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> mysql to hbase, how to custom row-family?


Copy link to this message
-
Re: mysql to hbase, how to custom row-family?
Hi Robber,
I'm glad to hear that you were able to find workaround for your use case! It seems that the HBase import wasn't working as expected for you when using --columns argument. If my understanding is correct, would you mind filling a JIRA [1], so that it can be fixed in future releases?

Jarcec

Links:
1: https://issues.apache.org/jira/browse/SQOOP

On Tue, May 21, 2013 at 01:34:01AM +0800, Robber Phex wrote:
> This is data in mysql table:
> +----+------+------+-------------+-------+
> | id | name | age  | email       | title |
> +----+------+------+-------------+-------+
> |  1 | aaaa |   20 | [EMAIL PROTECTED] | A     |
> |  2 | bbbb |   21 | [EMAIL PROTECTED] | B     |
> |  3 | cccc |   21 | [EMAIL PROTECTED] | A     |
> +----+------+------+-------------+-------+
>
> and, I want data in hbase:
> ROW
> COLUMN+CELL
>  1                    column=baseinfo:age, timestamp=1369041765578,
> value=20
>  1                    column=baseinfo:name, timestamp=1369041765578,
> value=aaaa
>  1                    column=workinfo:email, timestamp=1369041967157,value> [EMAIL PROTECTED]
>  1                    column=workinfo:title, timestamp=1369041967157,
> value=A
>  2                    column=baseinfo:age, timestamp=1369041778521,
> value=21
>  2                    column=baseinfo:name, timestamp=1369041778521,
> value=bbbb
>  2                    column=workinfo:email, timestamp=1369041970526, value> [EMAIL PROTECTED]
>  2                    column=workinfo:title, timestamp=1369041970526,
> value=B
>  3                    column=baseinfo:age, timestamp=1369041778521,
> value=21
>  3                    column=baseinfo:name, timestamp=1369041778521,
> value=cccc
>  3                    column=workinfo:email, timestamp=1369041970526, value> [EMAIL PROTECTED]
>  3                    column=workinfo:title, timestamp=1369041970526,
> value=A
> 3 row(s) in 1.1040 seconds
>
> I use two commonds:
> #name,age to baseinfo
> sqoop import --connect jdbc:mysql://h2/test --username hiveuser --password
> password \
>   --query 'SELECT id,name,age FROM testtable WHERE $CONDITIONS' \
>   --hbase-create-table --hbase-table hbasetable \
>   --column-family baseinfo --hbase-row-key id --split-by id
>
> #email,title to workinfo
> sqoop import --connect jdbc:mysql://h2/test --username hiveuser --password
> password \
>   --query 'SELECT id,email,title FROM testtable WHERE $CONDITIONS' \
>   --hbase-create-table --hbase-table hbasetable \
>   --column-family workinfo --hbase-row-key id --split-by id
>
> Anyone have more simple, easy-understand way?
>
> On Tue, May 21, 2013 at 12:55 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > Hi Robber,
> > Sqoop won't import rows where all columns that should be inserted into
> > cell contains NULL. Would you mind taking a look if in your case the name
> > and age are NULL? Another possibility to set the property
> > sqoop.hbase.add.row.key. You might find more information about this
> > property in SQOOP-870 [1] or in our user guide [1].
> >
> > Jarcec
> >
> > Links:
> > 1: https://issues.apache.org/jira/browse/SQOOP-870
> > 2:
> > http://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html#_additional_import_configuration_properties
> >
> > On Mon, May 20, 2013 at 11:02:56PM +0800, Robber Phex wrote:
> > > I think I can do it:
> > >
> > > HBase                       MySQL Columns
> > >
> > > row key    <-----------     id
> > > baseinfo   <-----------     name
> > >                  `-<---     age
> > >
> > > But, I use following command import zero record.
> > >
> > > sqoop import --connect jdbc:mysql://h2/test --username hiveuser
> > --password
> > > password --table testtable --hbase-table hbasetable --column-family
> > > baseinfo --columns "name,age" --hbase-row-key id
> > >
> > > Or,you means:
> > >
> > > sqoop import --connect jdbc:mysql://h2/test --username hiveuser
> > --password
> > > password --table testtable --hbase-table hbasetable --column-family name
> > > --hbase-row-key id
> > >
> > > sqoop import --connect jdbc:mysql://h2/test --username hiveuser
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB