Koert Kuipers 2012-02-08, 16:04
As you are creating dir/sub dirs using mapreduce jobs out of hive, hive is unaware of these sub dirs. There is no other way in such cases other than an add partition DDL to register the dir with a hive partition.
If you are using oozie or shell to trigger your jobs,you can accomplish it as
-use java to come up with the correct add partition statement and write those statement(s) into a file
-execute the file using hive -f <fileName>
Hope it helps!..
Bejoy K S
From handheld, Please excuse typos.
From: Koert Kuipers <[EMAIL PROTECTED]>
Date: Wed, 8 Feb 2012 11:04:18
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: external partitioned table
we have an external partitioned table in hive.
we add to this table by having map-reduce jobs (so not from hive) create
new subdirectories with the right format (partitionid=partitionvalue).
however hive doesn't pick them up automatically. we have to go into hive
shell and run "alter table sometable add partition
(partitionid=partitionvalue)". to make matter worse hive doesnt really lend
itself to running such an add-partition-operation from java (or for that
matter: hive doesn't lend itself to any easy programmatic manipulations...
grrr. but i will stop now before i go on a a rant).
any suggestions how to approach this? thanks!
Mark Grover 2012-02-08, 17:23
Roberto Congiu 2012-02-08, 19:18