Is it possible to instruct hive to get only specific files from a
partitioned external table?
For example I have the following directory structure

data/dd=2012-12-31/a1.txt
data/dd=2012-12-31/a2.txt
data/dd=2012-12-31/a3.txt
data/dd=2012-12-31/a4.txt

data/dd=2012-12-31/b1.txt
data/dd=2012-12-31/b2.txt
data/dd=2012-12-31/b2.txt

Is it possible to add 2012-12-31 as a partition and tell hive to load only
the a* files to the table?
Thanks,

If all files are in same partition then they satisfy condition of same
value as partion column .

You cannot do with hive but can have one intermediate table and then to
move required files using glob pattern

-----------
Sent from Mobile , short and crisp.
On 07-Jan-2013 1:07 AM, "Oded Poncz" <[EMAIL PROTECTED]> wrote:

> Is it possible to instruct hive to get only specific files from a
> partitioned external table?
> For example I have the following directory structure
>
> data/dd=2012-12-31/a1.txt
> data/dd=2012-12-31/a2.txt
> data/dd=2012-12-31/a3.txt
> data/dd=2012-12-31/a4.txt
>
> data/dd=2012-12-31/b1.txt
> data/dd=2012-12-31/b2.txt
> data/dd=2012-12-31/b2.txt
>
> Is it possible to add 2012-12-31 as a partition and tell hive to load only
> the a* files to the table?
> Thanks,
>

Hi Oded

If you have created the directories manually that would come visible to the hive table only if the partitions/ sub dirs are added to the meta data using
'ALTER TABLE ... ADD PARTITION' .
Partitions are not retrieved implicitly into hive tabe even if you have a proper sub dir structure.

Similarly if you don't need a particular partition on your table permanently you can always delete them using the alter table command.

If you are intending to use a particular partition alone in your query no need to alter the partitions. Just append a where clause to the query that has scope only on the required partitions.

Hope this helps.

Regards
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: Oded Poncz <[EMAIL PROTECTED]>
Date: Sun, 6 Jan 2013 16:07:26
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: External table with partitions

Is it possible to instruct hive to get only specific files from a
partitioned external table?
For example I have the following directory structure

data/dd=2012-12-31/a1.txt
data/dd=2012-12-31/a2.txt
data/dd=2012-12-31/a3.txt
data/dd=2012-12-31/a4.txt

data/dd=2012-12-31/b1.txt
data/dd=2012-12-31/b2.txt
data/dd=2012-12-31/b2.txt

Is it possible to add 2012-12-31 as a partition and tell hive to load only
the a* files to the table?
Thanks,

Sorry, I din understand your query on first look through.

Like Jagat said, you may need to go with a temp table for this.

Do a hadoop fs -cp ../../a.* <destn dir>

Create a external table with location as 'destn dir'.

CREATE EXERNAL TABLE <tmp tble name> LIKE <src table name> LOCATION '' ;

NB: I just gave the syntax from memory. please check the syntax in hive user guide.
Regards
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: [EMAIL PROTECTED]
Date: Sun, 6 Jan 2013 14:39:45
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: External table with partitions

Hi Oded

If you have created the directories manually that would come visible to the hive table only if the partitions/ sub dirs are added to the meta data using
'ALTER TABLE ... ADD PARTITION' .
Partitions are not retrieved implicitly into hive tabe even if you have a proper sub dir structure.

Similarly if you don't need a particular partition on your table permanently you can always delete them using the alter table command.

If you are intending to use a particular partition alone in your query no need to alter the partitions. Just append a where clause to the query that has scope only on the required partitions.

Hope this helps.

Regards
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: Oded Poncz <[EMAIL PROTECTED]>
Date: Sun, 6 Jan 2013 16:07:26
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: External table with partitions

Is it possible to instruct hive to get only specific files from a
partitioned external table?
For example I have the following directory structure

data/dd=2012-12-31/a1.txt
data/dd=2012-12-31/a2.txt
data/dd=2012-12-31/a3.txt
data/dd=2012-12-31/a4.txt

data/dd=2012-12-31/b1.txt
data/dd=2012-12-31/b2.txt
data/dd=2012-12-31/b2.txt

Is it possible to add 2012-12-31 as a partition and tell hive to load only
the a* files to the table?
Thanks,