Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> File Path and Partition names


+
carla.staeben@... 2012-10-02, 10:55
Copy link to this message
-
Re: File Path and Partition names
Hi Carla

If you like to have your custom directory structure for your  partitions. You can create dirs in hdfs of your choice , load data into them (If from another hive table then you can use 'Insert Overwrite Directory..' To populate an hdfs dir). Now you need to register this dir as a new partition on to required table using

'Alter Table Add Parition ...'
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: <[EMAIL PROTECTED]>
Date: Tue, 2 Oct 2012 10:55:19
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: File Path and Partition names

Quick question about using hive to create new hdfs file paths.

Generally speaking, we like to keep our data files with a path similar to

Dataset/year/month/day/hour

I need to create a new table in hive and populate it with data from a different dataset, using a HiveQL query.  If I do this:
CREATE EXTERNAL TABLE IF NOT EXISTS     new_table

(field1 string
,field2 string
,field3 string
)
partitioned by (reg_yr string, reg_mon string, reg_day string, reg_hour string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE  ;

And then do an insert overwrite into, I end up with this path in hdfs:

Dataset/reg_year=2012/reg_mon=10/reg_day=02/reg_hour=07

Is there an *easy* way to remove the partition name from the creation of the hdfs path?

Thanks
Carla

+
carla.staeben@... 2012-10-02, 12:56
+
Doug Houck 2012-10-02, 13:10
+
carla.staeben@... 2012-10-02, 13:16