Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> Re: Review Request 12050: HIVE-3756 (LOAD DATA does not honor permission inheritance)


Copy link to this message
-
Re: Review Request 12050: HIVE-3756 (LOAD DATA does not honor permission inheritance)

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12050/
-----------------------------------------------------------

(Updated July 19, 2013, 6:55 p.m.)
Review request for hive, Ashutosh Chauhan and Sushanth Sowmyan.
Bugs: HIVE- and HIVE-3756
    https://issues.apache.org/jira/browse/HIVE-
    https://issues.apache.org/jira/browse/HIVE-3756
Repository: hive-git
Description
-------

Problems:
1. When doing load data or insert overwrite to a table, the data files under database/table directory could not inherit their parent's permissions (i.e. group) as described in HIVE-3756.
2. Beside the group issue, the read/write permission mode is also not inherited
3. Same problem affects the partition files (see HIVE-3094)

Cause:
The task results (from load data or insert overwrite) are initially stored in scratchdir and then loaded under warehouse table directory. FileSystem.rename is used in this step (e.g. LoadTable/LoadPartition) to move the dirs/files but it preserves their permissions (including group and mode) which are determined by scratchdir permission or umask. If the scratchdir has different permissions from those of warehouse table directories, the problem occurs.

Solution:
After the FileSystem.rename is called, changing all renamed (moved) files/dirs to their destination parents' permissions if needed (say if hive.warehouse.subdir.inherit.perms is true). Here I introduced a new method renameFile doing both rename and permission. It replaces the FileSystem.rename used in LoadTable/LoadPartition. I do not replace rename used to move files/dirs under same scratchdir in the middle of task processing. It looks to me not necessary since they are temp files and also probably access protected by top scratchdir mode 700 (HIVE-4487).
Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 87a584d

Diff: https://reviews.apache.org/r/12050/diff/
Testing
-------

The following cases tested that all created subdirs/files inherit their parents' permission mode and group in : 1). create database; 2). create table; 3). load data; 4) insert overwrite; 5) partitions.
{code}
hive> dfs -ls -d /user/tester1/hive;                                                                      
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:20 /user/tester1/hive

hive> create database tester1 COMMENT 'Database for user tester1' LOCATION '/user/tester1/hive/tester1.db';
hive> dfs -ls -R /user/tester1/hive;                                                                        
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:21 /user/tester1/hive/tester1.db

hive> use tester1;
hive>  create table tester1.tst1(col1 int, col2 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE;
hive> dfs -ls -R /user/tester1/hive;                                                                                    
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:22 /user/tester1/hive/tester1.db
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:22 /user/tester1/hive/tester1.db/tst1

hive>  load data local inpath '/home/tester1/tst1.input' into table tst1;                                              
hive> dfs -ls -R /user/tester1/hive;                                    
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:22 /user/tester1/hive/tester1.db
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:23 /user/tester1/hive/tester1.db/tst1
-rw-rw----   3 tester1 testgroup123        168 2013-06-22 13:23 /user/tester1/hive/tester1.db/tst1/tst1.input

hive> create table tester1.tst2(col1 int, col2 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS SEQUENCEFILE;
hive> dfs -ls -R /user/tester1/hive;                                                                                    
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:24 /user/tester1/hive/tester1.db
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:23 /user/tester1/hive/tester1.db/tst1
-rw-rw----   3 tester1 testgroup123        168 2013-06-22 13:23 /user/tester1/hive/tester1.db/tst1/tst1.input
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:24 /user/tester1/hive/tester1.db/tst2

hive> insert overwrite table tst2 select * from tst1;
hive> dfs -ls -R /user/tester1/hive;                
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:25 /user/tester1/hive/tester1.db
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:23 /user/tester1/hive/tester1.db/tst1
-rw-rw----   3 tester1 testgroup123        168 2013-06-22 13:23 /user/tester1/hive/tester1.db/tst1/tst1.input
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:25 /user/tester1/hive/tester1.db/tst2
-rw-rw----   3 tester1 testgroup123        291 2013-06-22 13:25 /user/tester1/hive/tester1.db/tst2/000000_0

hive> create table tester1.tst3(col2 string) PARTITIONED BY (col1 int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE;
hive> dfs -ls -R /user/tester1/hive;                                                                                    
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:27 /user/tester1/hive/tester1.db
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:23 /user/tester1/hive/tester1.db/tst1
-rw-rw----   3 tester1 testgroup123        168 2013-06-22 13:23 /user/tester1/hive/tester1.db/tst1/tst1.input
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:25 /user/tester1/hive/tester1.db/tst2
-rw-rw----   3 tester1 testgroup123        291 2013-06-22 13:25 /user/tester1/hive/tester1.db/tst2/000000_0
drwxrwx---   - tester1 testgroup123          0 2013-06-22 13:27 /user/tester1/hive/tester1.db/tst3

hive> set hive.exec.dynamic.partition.mode=nonstrict;                                                                  
hive> insert overwrite table tester1.tst3 partition (col1)