Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop not picking up immediate changes


Copy link to this message
-
Re: Sqoop not picking up immediate changes
Hi Blues,
would you mind sharing details about your use case? Table schemas, exact commands (both on database and in command line) and associated logs?

Wild guess - when you are changing the rows in the database, are you committing the ongoing transaction? Sqoop will create a new connection with new transaction, so due to ACID it won't pick up any uncommitted changes.

Jarcec

On Tue, Nov 12, 2013 at 10:36:10PM -0800, burberry blues wrote:
> Hi Team,
>
> I am having a problem with following scenario.
>
> In Db i update a column1 of a row and the column 2 got modified with
> current timestamp.
> But when i try to import those changes through sqoop using --incremental
> lastmodified --check-column column2 --last-value <less than current
> date>,it shows 0 records imported which are changed.
>
> There are changes in the DB but sqoop qorks as if it couldnt find the
> updated once and still pointing to the old records.
>
> i.e Before updating i have 3 records with date as 10th Nov,i asked sqoop to
> import records after 9th Nov. It imports all 3 records.
> Now i change 1 row and date is updated to 12 Nov. Immediate I ask sqoop to
> import records after 11th Nov .But it imports 0 records now.If i run the
> same import with date as 9th nov again it works fine and also* give me
> duplicate records*.
>
> Please help me in this issue at the earliest.
>
> Thanks,
> Blues
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB