Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> SQL Server sqoop export very slow


+
Nitin kak 2013-02-04, 19:51
Copy link to this message
-
Re: SQL Server sqoop export very slow
Hi Nitin,
I'm afraid that Microsoft SQL Server connector do not support direct mode and thus the --direct parameter should not make any difference. Based on my personal experience the Microsoft's connector have comparable performance with the build-in one and thus I'm not expecting any change with regards to performance.

It seems that in your case you are using three threads (mappers) writing data simultaneously into database using JDBC interface. I would say that 1 hour for 2 GB data set is quite much. I would suggest to taking a look on SQL Server performance metrics during the export (IO, CPU, ...) to see where is the bottleneck.

Jarcec

On Mon, Feb 04, 2013 at 02:51:17PM -0500, Nitin kak wrote:
> Hi All,
>
>     I am working on the sqoop export to SQL Server. It took like 1 hr to
> export 2 GB of data. Three mappers had spawned. I am using Default SQL
> Server Connector provided by cloudera(Couldn't use MS Sql Server Connector
> because of a bug in export). I was using --direct clause. Doesnt seem to
> have much effect. Any clues why its so slow? Would the performance improve
> once I start using MS SQLServerConnector?
>
> Regards,
> Nitin
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB