Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> SQOOP 1.3.0 questions using SQL Server

Copy link to this message
Re: SQOOP 1.3.0 questions using SQL Server
Hi Artem,
this is actually very helpful log fragment. Sqoop 1.4.2 do support the NVARCHAR data type for import/export (which was not the case for Sqoop 1.3.0), however Sqoop do not have special Splitter for NVARCHAR type. Which means that such column can't be used for parameter --split-by nor explicitly nor implicitly. One possible workaround is to use parameter "--num-mappers 1" that will limit number of active mappers to 1 and thus there will be no need for splitter. However such solution will obviously have performance issues as there won't be any parallel data movement. Are you able by any chance add some number based column or something?

I believe that this exception is not much helpful, so I've filled SQOOP-652 [1] on your behalf to improve that for more reasonable error message. I've also looked around and I believe that we could reuse TextSplitter that is already there to support splitting by N(LONG)(VAR)CHAR column. I've filled SQOOP-653 [2] to keep track of that. I've marked both issues with "newbie" label as I believe that they should be rather easy to resolve.


1: https://issues.apache.org/jira/browse/SQOOP-652
2: https://issues.apache.org/jira/browse/SQOOP-653

On Fri, Oct 26, 2012 at 09:48:45PM +0000, Artem Ervits wrote:
> Don't pay attention to error that this table has two primary keys, I just want to know whether nvarchar still works.
> [xxxxx@SERVERNAME ~]$ clear
> [xxxxx@SERVERNAME ~]$ $SQOOP_HOME/bin/sqoop import-all-tables --connect "jdbc:sqlserver://dbserver;username=xxxx;password=xxxxx;database=xxxxx" --hive-import --create-hive-table -compress  -verbose
> 12/10/26 17:39:52 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 12/10/26 17:39:52 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
> 12/10/26 17:39:52 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
> 12/10/26 17:39:52 ERROR sqoop.ConnFactory: Error loading ManagerFactory information from file /usr/local/sqoop/conf/managers.d/mssqoop-sqlserver: java.io.IOException: the content of connector file must be in form of key=value
>         at org.apache.sqoop.ConnFactory.addManagersFromFile(ConnFactory.java:219)
>         at org.apache.sqoop.ConnFactory.loadManagersFromConfDir(ConnFactory.java:294)
>         at org.apache.sqoop.ConnFactory.instantiateFactories(ConnFactory.java:85)
>         at org.apache.sqoop.ConnFactory.<init>(ConnFactory.java:62)
>         at com.cloudera.sqoop.ConnFactory.<init>(ConnFactory.java:36)
>         at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:200)
>         at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:83)
>         at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:48)
>         at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>         at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>         at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> 12/10/26 17:39:52 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/10/26 17:39:52 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/10/26 17:39:52 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:sqlserver:
> 12/10/26 17:39:52 INFO manager.SqlManager: Using default fetchSize of 1000
> 12/10/26 17:39:52 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.SQLServerManager@380e28b9
> 12/10/26 17:39:52 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
> 12/10/26 17:39:53 INFO tool.CodeGenTool: Beginning code generation
> 12/10/26 17:39:53 DEBUG manager.SqlManager: Using fetchSize for next query: 1000
> 12/10/26 17:39:53 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM [TABLENAME] AS t WHERE 1=0