Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> SQOOP 1.3.0 questions using SQL Server


+
Artem Ervits 2012-10-12, 17:29
+
Jarek Jarcec Cecho 2012-10-12, 17:50
+
Artem Ervits 2012-10-12, 18:01
+
Jarek Jarcec Cecho 2012-10-12, 18:08
+
Artem Ervits 2012-10-12, 18:16
+
Chalcy 2012-10-12, 20:18
+
Artem Ervits 2012-10-12, 21:16
+
Artem Ervits 2012-10-15, 19:25
+
Jarek Jarcec Cecho 2012-10-15, 20:33
+
Artem Ervits 2012-10-26, 21:09
+
Jarek Jarcec Cecho 2012-10-26, 21:17
+
Artem Ervits 2012-10-26, 21:48
Copy link to this message
-
Re: SQOOP 1.3.0 questions using SQL Server
Hi Artem,
this is actually very helpful log fragment. Sqoop 1.4.2 do support the NVARCHAR data type for import/export (which was not the case for Sqoop 1.3.0), however Sqoop do not have special Splitter for NVARCHAR type. Which means that such column can't be used for parameter --split-by nor explicitly nor implicitly. One possible workaround is to use parameter "--num-mappers 1" that will limit number of active mappers to 1 and thus there will be no need for splitter. However such solution will obviously have performance issues as there won't be any parallel data movement. Are you able by any chance add some number based column or something?

I believe that this exception is not much helpful, so I've filled SQOOP-652 [1] on your behalf to improve that for more reasonable error message. I've also looked around and I believe that we could reuse TextSplitter that is already there to support splitting by N(LONG)(VAR)CHAR column. I've filled SQOOP-653 [2] to keep track of that. I've marked both issues with "newbie" label as I believe that they should be rather easy to resolve.

Jarcec

Links:
1: https://issues.apache.org/jira/browse/SQOOP-652
2: https://issues.apache.org/jira/browse/SQOOP-653

On Fri, Oct 26, 2012 at 09:48:45PM +0000, Artem Ervits wrote:
> Don't pay attention to error that this table has two primary keys, I just want to know whether nvarchar still works.
>
> [xxxxx@SERVERNAME ~]$ clear
> [xxxxx@SERVERNAME ~]$ $SQOOP_HOME/bin/sqoop import-all-tables --connect "jdbc:sqlserver://dbserver;username=xxxx;password=xxxxx;database=xxxxx" --hive-import --create-hive-table -compress  -verbose
> 12/10/26 17:39:52 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 12/10/26 17:39:52 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
> 12/10/26 17:39:52 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
> 12/10/26 17:39:52 ERROR sqoop.ConnFactory: Error loading ManagerFactory information from file /usr/local/sqoop/conf/managers.d/mssqoop-sqlserver: java.io.IOException: the content of connector file must be in form of key=value
>         at org.apache.sqoop.ConnFactory.addManagersFromFile(ConnFactory.java:219)
>         at org.apache.sqoop.ConnFactory.loadManagersFromConfDir(ConnFactory.java:294)
>         at org.apache.sqoop.ConnFactory.instantiateFactories(ConnFactory.java:85)
>         at org.apache.sqoop.ConnFactory.<init>(ConnFactory.java:62)
>         at com.cloudera.sqoop.ConnFactory.<init>(ConnFactory.java:36)
>         at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:200)
>         at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:83)
>         at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:48)
>         at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>         at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>         at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
>
> 12/10/26 17:39:52 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/10/26 17:39:52 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/10/26 17:39:52 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:sqlserver:
> 12/10/26 17:39:52 INFO manager.SqlManager: Using default fetchSize of 1000
> 12/10/26 17:39:52 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.SQLServerManager@380e28b9
> 12/10/26 17:39:52 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
> 12/10/26 17:39:53 INFO tool.CodeGenTool: Beginning code generation
> 12/10/26 17:39:53 DEBUG manager.SqlManager: Using fetchSize for next query: 1000
> 12/10/26 17:39:53 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM [TABLENAME] AS t WHERE 1=0
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB