Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Hive External Table issue


+
sanjeev sagar 2013-06-20, 17:22
+
Nitin Pawar 2013-06-20, 17:24
+
sanjeev sagar 2013-06-20, 17:27
+
Nitin Pawar 2013-06-20, 17:30
+
sanjeev sagar 2013-06-20, 18:03
Copy link to this message
-
Re: Hive External Table issue
1. I was under the impression that you cannot refer the table location to a
file. But, it looks like it works. Please see the discussion in the thread
 http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%
[EMAIL PROTECTED]%3e

2. It there are more than one file in the directory, your query gets the
data from all the files in that directory.

In your case, the regex may not be parsing the data properly.

Regards,
Ramki.
On Thu, Jun 20, 2013 at 11:03 AM, sanjeev sagar <[EMAIL PROTECTED]>wrote:

> Two issues:
>
> 1. I've created external tables in hive based on file location before and
> it work without any issue. It don't have to be a directory.
>
> 2. If there are more than one file in the directory, and you create
> external table based on directory then how the table knows that which file
> it need to look for the data?
>
> I tried to create the table based on directory, it created the table but
> all the rows were NULL.
>
> -Sanjeev
>
>
> On Thu, Jun 20, 2013 at 10:30 AM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> in hive when you create table and use the location to refer hdfs path,
>> that path is supposed to be a directory.
>> If the directory is not existing it will try to create it and if its a
>> file it will throw an error as its not a directory
>>
>> thats the error you are getting that location you referred is a file.
>> Change it to the directory and see if that works for you
>>
>>
>> On Thu, Jun 20, 2013 at 10:57 PM, sanjeev sagar <[EMAIL PROTECTED]>wrote:
>>
>>> I did mention in my mail the hdfs file exists in that location. See below
>>>
>>> In HDFS: file exists
>>>
>>>
>>>
>>> hadoop fs -ls
>>>
>>> /user/flume/events/request_logs/
>>> ar1.vgs.mypoints.com/13-06-13/FlumeData.1371144648033
>>>
>>> Found 1 items
>>>
>>> -rw-r--r--   3 hdfs supergroup 2242037226 2013-06-13 11:14
>>>
>>> /user/flume/events/request_logs/
>>> ar1.vgs.mypoints.com/13-06-13/FlumeData.1371144648033
>>>
>>> so the directory and file both exists.
>>>
>>>
>>> On Thu, Jun 20, 2013 at 10:24 AM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>>
>>>> MetaException(message:hdfs://
>>>> h1.vgs.mypoints.com:8020/user/flume/events/request_logs/ar1.vgs.mypoints.com/13-06-13/FlumeData.1371144648033
>>>>
>>>> is not a directory or unable to create one)
>>>>
>>>>
>>>> it clearly says its not a directory. Point to the dictory and it will
>>>> work
>>>>
>>>>
>>>> On Thu, Jun 20, 2013 at 10:52 PM, sanjeev sagar <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hello Everyone, I'm running into the following Hive external table
>>>>> issue.
>>>>>
>>>>>
>>>>>
>>>>> hive> CREATE EXTERNAL TABLE access(
>>>>>
>>>>>      >       host STRING,
>>>>>
>>>>>      >       identity STRING,
>>>>>
>>>>>      >       user STRING,
>>>>>
>>>>>      >       time STRING,
>>>>>
>>>>>      >       request STRING,
>>>>>
>>>>>      >       status STRING,
>>>>>
>>>>>      >       size STRING,
>>>>>
>>>>>      >       referer STRING,
>>>>>
>>>>>      >       agent STRING)
>>>>>
>>>>>      >       ROW FORMAT SERDE
>>>>>
>>>>> 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
>>>>>
>>>>>      >       WITH SERDEPROPERTIES (
>>>>>
>>>>>      >      "input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\])
>>>>>
>>>>> ([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\") ([^
>>>>> \"]*|\"[^\"]*\"))?",
>>>>>
>>>>>      >       "output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s
>>>>>
>>>>> %7$s %8$s %9$s"
>>>>>
>>>>>      >       )
>>>>>
>>>>>      >       STORED AS TEXTFILE
>>>>>
>>>>>      >       LOCATION
>>>>>
>>>>> '/user/flume/events/request_logs/
>>>>> ar1.vgs.mypoints.com/13-06-13/FlumeData.1371144648033';
>>>>>
>>>>> FAILED: Error in metadata:
>>>>>
>>>>> MetaException(message:hdfs://
>>>>> h1.vgs.mypoints.com:8020/user/flume/events/request_logs/ar1.vgs.mypoints.com/13-06-13/FlumeData.1371144648033
>>>>>
>>>>> is not a directory or unable to create one)
>>>>>
>>>>> FAILED: Execution Error, return code 1 from
+
Nitin Pawar 2013-06-20, 18:11
+
Nitin Pawar 2013-06-20, 18:14
+
Ramki Palle 2013-06-20, 18:34
+
Nitin Pawar 2013-06-20, 18:55
+
Stephen Sprague 2013-06-20, 19:19