Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> File Path and Partition names


Copy link to this message
-
Re: File Path and Partition names
Hi Carla

If you like to have your custom directory structure for your  partitions. You can create dirs in hdfs of your choice , load data into them (If from another hive table then you can use 'Insert Overwrite Directory..' To populate an hdfs dir). Now you need to register this dir as a new partition on to required table using

'Alter Table Add Parition ...'
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: <[EMAIL PROTECTED]>
Date: Tue, 2 Oct 2012 10:55:19
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: File Path and Partition names

Quick question about using hive to create new hdfs file paths.

Generally speaking, we like to keep our data files with a path similar to

Dataset/year/month/day/hour

I need to create a new table in hive and populate it with data from a different dataset, using a HiveQL query.  If I do this:
CREATE EXTERNAL TABLE IF NOT EXISTS     new_table

(field1 string
,field2 string
,field3 string
)
partitioned by (reg_yr string, reg_mon string, reg_day string, reg_hour string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE  ;

And then do an insert overwrite into, I end up with this path in hdfs:

Dataset/reg_year=2012/reg_mon=10/reg_day=02/reg_hour=07

Is there an *easy* way to remove the partition name from the creation of the hdfs path?

Thanks
Carla

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB