Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> CREATE EXTERNAL TABLE Fails on Some Directories


Copy link to this message
-
Re: CREATE EXTERNAL TABLE Fails on Some Directories

Hi Joseph

There are differences in the following ls commands

cloudera@localhost data]$ hdfs dfs -ls /715

This would list out all the contents in /715 in hdfs, if it is a dir

Found 1 items
-rw-r--r--   1 cloudera supergroup    7853975 2013-02-14 17:03 /715

The output clearly defines it is file as d is missing as the first char

[cloudera@localhost data]$ hdfs dfs -ls 715

This lists the dir 715 in your user's hdfs home dir. If your user is cloudera then usually your home dir might be /userdata/cloudera/ so in effect the dir listed is /userdata/cloudera/715
Regards
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: Joseph D Antoni <[EMAIL PROTECTED]>
Date: Fri, 15 Feb 2013 08:55:50
To: [EMAIL PROTECTED]<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories

Not sure--I just truncated the file list from the ls--that was the first file (just obfuscated the name)

The command I used to create the directories was:

hdfs dfs -mkdir 715
then
hdfs dfs -put myfile.csv 715

[cloudera@localhost data]$ hdfs dfs -ls /715
Found 1 items
-rw-r--r--   1 cloudera supergroup    7853975 2013-02-14 17:03 /715
[cloudera@localhost data]$ hdfs dfs -ls 715
Found 13 items
-rw-r--r--   1 cloudera cloudera    7853975 2013-02-15 00:41 715/40-file.csv

Thanks

________________________________
 From: Dean Wampler <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Joseph D Antoni <[EMAIL PROTECTED]>
Sent: Friday, February 15, 2013 11:50 AM
Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories
 

Something's odd about this output; why is there no / in front of 715? I always get the full path when I run a -ls command. I would expect either:

/715/file.csv
or
/user/<me>/715/file.csv

Or is that what you meant by "(didn't leave rest of ls results)"?

dean
On Fri, Feb 15, 2013 at 10:45 AM, Joseph D Antoni <[EMAIL PROTECTED]> wrote:

[cloudera@localhost data]$ hdfs dfs -ls 715
>Found 13 items
>-rw-r--r--   1 cloudera cloudera    7853975 2013-02-15 00:41 715/file.csv (didn't leave rest of ls results)
>
>
>Thanks on the directory--wasn't clear on that..
>
>Joey
>
>
>
>
>
>
>
>
>________________________________
> From: Dean Wampler <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]; Joseph D Antoni <[EMAIL PROTECTED]>
>Sent: Friday, February 15, 2013 11:37 AM
>Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories
>
>
>
>You confirmed that 715 is an actual directory? It didn't become a file by accident?
>
>
>By the way, you don't need to include the file name in the LOCATION. It will read all the files in the directory.
>
>
>dean
>
>
>On Fri, Feb 15, 2013 at 10:29 AM, Joseph D Antoni <[EMAIL PROTECTED]> wrote:
>
>I'm trying to create a series of external tables for a time series of data (using the prebuilt Cloudera VM).
>>
>>
>>The directory structure in HDFS is as such:
>>
>>
>>/711
>>/712
>>/713
>>/714
>>/715
>>/716
>>/717
>>
>>
>>Each directory contains the same set of files, from a different day. They were all put into HDFS using the following script:
>>
>>
>>for i in *;do hdfs dfs -put $i in $dir;done
>>
>>
>>They all show up with the same ownership/perms in HDFS.
>>
>>
>>Going into Hive to build the tables, I built a set of scripts to do the loads--then did a sed (changing 711 to 712,713, etc) to a file for each day. All of my loads work, EXCEPT for 715 and 716. 
>>
>>
>>Script is as follows:
>>
>>
>>create external table 715_table_name
>>(col1 string,
>>col2 string)
>>row format
>>delimited fields terminated by ','
>>lines terminated by '\n'
>>stored as textfile
>>location '/715/file.csv';
>>
>>
>>This is failing with:
>>
>>
>>Error in Metadata MetaException(message:Got except: org.apache.hadoop.fs.FileAlreadyExistsException Parent Path is not a directory: /715 715...
>>
>>
>>Like I mentioned it works for all of the other directories, except 715 and 716. Thoughts on troubleshooting path?
>>
>>
>>Thanks
Dean Wampler, Ph.D.
thinkbiganalytics.com
+1-312-339-1330