Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hive External Table issue


Copy link to this message
-
Re: Hive External Table issue
Nitin Pawar 2013-06-20, 18:55
Ramki,

I was going through that thread before as Sanjeev said it worked so I was
doing some experiment as well.
As you I too had the impression that Hive tables are associated with
directories and as pointed out I was wrong.

Basically the idea of pointing a table to a file as mentioned on that
thread is kind of hack
create table without location
alter table to point to file

>From Mark's answer what he suggest is we can use virtual column
INPUT__FILE__NAME to select which file we want to use while querying in
case there are multiple files inside a directory and you just want to use a
specific one.

The bug I mentioned is for  files, having particular files from a directory
matching the regex. Not for the regex serde.

Correct my understanding if I got anything wrong
On Fri, Jun 21, 2013 at 12:04 AM, Ramki Palle <[EMAIL PROTECTED]> wrote:

> Nitin,
>
> Can you go through the thread with subject "S3/EMR Hive: Load contents of
> a single file"  on Tue, 26 Mar, 17:11> at
>
>
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/thread?1
>
>  This gives the whole discussion about the topic of table location
> pointing to a filename vs. directory.
>
> Can you give your insight from this discussion and the discussion you
> mentioned at stackoverflow link?
>
> Regards,
> Ramki.
>
>
>
> On Thu, Jun 20, 2013 at 11:14 AM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> Also see this JIRA
>> https://issues.apache.org/jira/browse/HIVE-951
>>
>> I think issue you are facing is due to the JIRA
>>
>>
>> On Thu, Jun 20, 2013 at 11:41 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>
>>> Mark has answered this before
>>>
>>> http://stackoverflow.com/questions/11269203/when-creating-an-external-table-in-hive-can-i-point-the-location-to-specific-fil
>>>
>>> If this link does not answer your question, do let us know
>>>
>>>
>>> On Thu, Jun 20, 2013 at 11:33 PM, sanjeev sagar <[EMAIL PROTECTED]
>>> > wrote:
>>>
>>>> Two issues:
>>>>
>>>> 1. I've created external tables in hive based on file location before
>>>> and it work without any issue. It don't have to be a directory.
>>>>
>>>> 2. If there are more than one file in the directory, and you create
>>>> external table based on directory then how the table knows that which file
>>>> it need to look for the data?
>>>>
>>>> I tried to create the table based on directory, it created the table
>>>> but all the rows were NULL.
>>>>
>>>> -Sanjeev
>>>>
>>>>
>>>> On Thu, Jun 20, 2013 at 10:30 AM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> in hive when you create table and use the location to refer hdfs path,
>>>>> that path is supposed to be a directory.
>>>>> If the directory is not existing it will try to create it and if its a
>>>>> file it will throw an error as its not a directory
>>>>>
>>>>> thats the error you are getting that location you referred is a file.
>>>>> Change it to the directory and see if that works for you
>>>>>
>>>>>
>>>>> On Thu, Jun 20, 2013 at 10:57 PM, sanjeev sagar <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> I did mention in my mail the hdfs file exists in that location. See
>>>>>> below
>>>>>>
>>>>>> In HDFS: file exists
>>>>>>
>>>>>>
>>>>>>
>>>>>> hadoop fs -ls
>>>>>>
>>>>>> /user/flume/events/request_logs/
>>>>>> ar1.vgs.mypoints.com/13-06-13/FlumeData.1371144648033
>>>>>>
>>>>>> Found 1 items
>>>>>>
>>>>>> -rw-r--r--   3 hdfs supergroup 2242037226 2013-06-13 11:14
>>>>>>
>>>>>> /user/flume/events/request_logs/
>>>>>> ar1.vgs.mypoints.com/13-06-13/FlumeData.1371144648033
>>>>>>
>>>>>> so the directory and file both exists.
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 20, 2013 at 10:24 AM, Nitin Pawar <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> MetaException(message:hdfs://
>>>>>>> h1.vgs.mypoints.com:8020/user/flume/events/request_logs/ar1.vgs.mypoints.com/13-06-13/FlumeData.1371144648033
>>>>>>>
>>>>>>> is not a directory or unable to create one)
>>>>>>>
>>>>>>>
>>>>>>> it clearly says its not a directory. Point to the dictory and it
Nitin Pawar