Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Sqoop export - incremental extracts


+
Sadananda Hegde 2012-10-26, 00:38
Copy link to this message
-
Re: Sqoop export - incremental extracts
Hi Sadu,
unfortunately Sqoop export is taking entire input directory (--export-dir) and simply exporting it's content to the external database/warehouse system. I'm afraid that there isn't more sophisticated way of doing "incremental" exports then using different hdfs directories for each "incremental" part.

If you could describe your use case, there might be other ways how to achieve similar results.

Jarcec

On Thu, Oct 25, 2012 at 07:38:21PM -0500, Sadananda Hegde wrote:
> Hello,
>
> I am exploring sqoop to send data from hadoop to EDW. I don't want to send
> the same data again and again. I need to identify the changes in HDFS and
> send only the data that has changed since my previous export. What is the
> best way to implement such incremental export logic?  I see that sqoop
> import has incremental logic option; but can't see it in export.
>
> Any recomendations / suggestions would greatly be appreciated.
>
> Thanks,
> Sadu
+
Sadananda Hegde 2012-10-26, 19:01
+
Jarek Jarcec Cecho 2012-10-26, 21:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB