Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> Review Request 16938: HIVE-6209 'LOAD DATA INPATH ... OVERWRITE ..' doesn't overwrite current data


Copy link to this message
-
Re: Review Request 16938: HIVE-6209 'LOAD DATA INPATH ... OVERWRITE ..' doesn't overwrite current data

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16938/#review32352
-----------------------------------------------------------

ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
<https://reviews.apache.org/r/16938/#comment61105>

    Can you please add a test case (.q file ) around these condition? This is in an important execution code path.
- Mohammad Islam
On Jan. 16, 2014, 1:45 a.m., Szehon Ho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16938/
> -----------------------------------------------------------
>
> (Updated Jan. 16, 2014, 1:45 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-6209
>     https://issues.apache.org/jira/browse/HIVE-6209
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> There was a wrong condition introduced in HIVE-3756, that prevented load data overwrite from working properly.  In these situations, destf == oldPath == /user/warehouse/hive/<tableName>, so -rmr was skipped on old data.
>
> Note that if file name was same, ie load data inpath '<path>' with same path repeatedly, it would work as the rename would overwrite the old data file.  But in this case, the filename is different.
>
> Other minor changes are trying to improve logging in this area to better diagnose the issues (for example file permission, etc).
>
>
> Diffs
> -----
>
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2fe86e1
>
> Diff: https://reviews.apache.org/r/16938/diff/
>
>
> Testing
> -------
>
> The primary concern was whether removing the directory in these scenarios would make the rename fail.  It should not due to fs.mkdirs call before, but I still verified the following scenarios:
>
> load/insert overwrite into table with partitions
> load/insert overwrite into table with buckets
>
>
> Thanks,
>
> Szehon Ho
>
>