I'm trying to create a series of external tables for a time series of data (using the prebuilt Cloudera VM).

The directory structure in HDFS is as such:

/711
/712
/713
/714
/715
/716
/717

Each directory contains the same set of files, from a different day. They were all put into HDFS using the following script:

for i in *;do hdfs dfs -put $i in $dir;done

They all show up with the same ownership/perms in HDFS.

Going into Hive to build the tables, I built a set of scripts to do the loads--then did a sed (changing 711 to 712,713, etc) to a file for each day. All of my loads work, EXCEPT for 715 and 716. 

Script is as follows:

create external table 715_table_name
(col1 string,
col2 string)
row format
delimited fields terminated by ','
lines terminated by '\n'
stored as textfile
location '/715/file.csv';

This is failing with:

Error in Metadata MetaException(message:Got except: org.apache.hadoop.fs.FileAlreadyExistsException Parent Path is not a directory: /715 715...

Like I mentioned it works for all of the other directories, except 715 and 716. Thoughts on troubleshooting path?

Thanks

Joey D'Antoni

You confirmed that 715 is an actual directory? It didn't become a file by
accident?

By the way, you don't need to include the file name in the LOCATION. It
will read all the files in the directory.

dean

On Fri, Feb 15, 2013 at 10:29 AM, Joseph D Antoni <[EMAIL PROTECTED]>wrote:

> I'm trying to create a series of external tables for a time series of data
> (using the prebuilt Cloudera VM).
>
> The directory structure in HDFS is as such:
>
> /711
> /712
> /713
> /714
> /715
> /716
> /717
>
> Each directory contains the same set of files, from a different day. They
> were all put into HDFS using the following script:
>
> for i in *;do hdfs dfs -put $i in $dir;done
>
> They all show up with the same ownership/perms in HDFS.
>
> Going into Hive to build the tables, I built a set of scripts to do the
> loads--then did a sed (changing 711 to 712,713, etc) to a file for each
> day. All of my loads work, EXCEPT for 715 and 716.
>
> Script is as follows:
>
> create external table 715_table_name
> (col1 string,
> col2 string)
> row format
> delimited fields terminated by ','
> lines terminated by '\n'
> stored as textfile
> location '/715/file.csv';
>
> This is failing with:
>
> Error in Metadata MetaException(message:Got except:
> org.apache.hadoop.fs.FileAlreadyExistsException Parent Path is not a
> directory: /715 715...
>
> Like I mentioned it works for all of the other directories, except 715 and
> 716. Thoughts on troubleshooting path?
>
> Thanks
>
> Joey D'Antoni
>

--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330

[cloudera@localhost data]$ hdfs dfs -ls 715
Found 13 items
-rw-r--r--   1 cloudera cloudera    7853975 2013-02-15 00:41 715/file.csv (didn't leave rest of ls results)
Thanks on the directory--wasn't clear on that..

Joey

________________________________
 From: Dean Wampler <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Joseph D Antoni <[EMAIL PROTECTED]>
Sent: Friday, February 15, 2013 11:37 AM
Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories
 

You confirmed that 715 is an actual directory? It didn't become a file by accident?

By the way, you don't need to include the file name in the LOCATION. It will read all the files in the directory.

dean
On Fri, Feb 15, 2013 at 10:29 AM, Joseph D Antoni <[EMAIL PROTECTED]> wrote:

I'm trying to create a series of external tables for a time series of data (using the prebuilt Cloudera VM).
>
>
>The directory structure in HDFS is as such:
>
>
>/711
>/712
>/713
>/714
>/715
>/716
>/717
>
>
>Each directory contains the same set of files, from a different day. They were all put into HDFS using the following script:
>
>
>for i in *;do hdfs dfs -put $i in $dir;done
>
>
>They all show up with the same ownership/perms in HDFS.
>
>
>Going into Hive to build the tables, I built a set of scripts to do the loads--then did a sed (changing 711 to 712,713, etc) to a file for each day. All of my loads work, EXCEPT for 715 and 716. 
>
>
>Script is as follows:
>
>
>create external table 715_table_name
>(col1 string,
>col2 string)
>row format
>delimited fields terminated by ','
>lines terminated by '\n'
>stored as textfile
>location '/715/file.csv';
>
>
>This is failing with:
>
>
>Error in Metadata MetaException(message:Got except: org.apache.hadoop.fs.FileAlreadyExistsException Parent Path is not a directory: /715 715...
>
>
>Like I mentioned it works for all of the other directories, except 715 and 716. Thoughts on troubleshooting path?
>
>
>Thanks
>
>
>Joey D'Antoni
--
Dean Wampler, Ph.D.
thinkbiganalytics.com
+1-312-339-1330

Something's odd about this output; why is there no / in front of 715? I
always get the full path when I run a -ls command. I would expect either:

/715/file.csv
or
/user/<me>/715/file.csv

Or is that what you meant by "(didn't leave rest of ls results)"?

dean

On Fri, Feb 15, 2013 at 10:45 AM, Joseph D Antoni <[EMAIL PROTECTED]>wrote:

> [cloudera@localhost data]$ hdfs dfs -ls 715
> Found 13 items
> -rw-r--r--   1 cloudera cloudera    7853975 2013-02-15 00:41 715/file.csv
> (didn't leave rest of ls results)
>
>
> Thanks on the directory--wasn't clear on that..
>
> Joey
>
>
>
>   ------------------------------
> *From:* Dean Wampler <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]; Joseph D Antoni <[EMAIL PROTECTED]>
> *Sent:* Friday, February 15, 2013 11:37 AM
> *Subject:* Re: CREATE EXTERNAL TABLE Fails on Some Directories
>
> You confirmed that 715 is an actual directory? It didn't become a file by
> accident?
>
> By the way, you don't need to include the file name in the LOCATION. It
> will read all the files in the directory.
>
> dean
>
> On Fri, Feb 15, 2013 at 10:29 AM, Joseph D Antoni <[EMAIL PROTECTED]>wrote:
>
> I'm trying to create a series of external tables for a time series of data
> (using the prebuilt Cloudera VM).
>
> The directory structure in HDFS is as such:
>
> /711
> /712
> /713
> /714
> /715
> /716
> /717
>
> Each directory contains the same set of files, from a different day. They
> were all put into HDFS using the following script:
>
> for i in *;do hdfs dfs -put $i in $dir;done
>
> They all show up with the same ownership/perms in HDFS.
>
> Going into Hive to build the tables, I built a set of scripts to do the
> loads--then did a sed (changing 711 to 712,713, etc) to a file for each
> day. All of my loads work, EXCEPT for 715 and 716.
>
> Script is as follows:
>
> create external table 715_table_name
> (col1 string,
> col2 string)
> row format
> delimited fields terminated by ','
> lines terminated by '\n'
> stored as textfile
> location '/715/file.csv';
>
> This is failing with:
>
> Error in Metadata MetaException(message:Got except:
> org.apache.hadoop.fs.FileAlreadyExistsException Parent Path is not a
> directory: /715 715...
>
> Like I mentioned it works for all of the other directories, except 715 and
> 716. Thoughts on troubleshooting path?
>
> Thanks
>
> Joey D'Antoni
>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
>
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330

Not sure--I just truncated the file list from the ls--that was the first file (just obfuscated the name)

The command I used to create the directories was:

hdfs dfs -mkdir 715
then
hdfs dfs -put myfile.csv 715

[cloudera@localhost data]$ hdfs dfs -ls /715
Found 1 items
-rw-r--r--   1 cloudera supergroup    7853975 2013-02-14 17:03 /715
[cloudera@localhost data]$ hdfs dfs -ls 715
Found 13 items
-rw-r--r--   1 cloudera cloudera    7853975 2013-02-15 00:41 715/40-file.csv

Thanks

________________________________
 From: Dean Wampler <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Joseph D Antoni <[EMAIL PROTECTED]>
Sent: Friday, February 15, 2013 11:50 AM
Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories
 

Something's odd about this output; why is there no / in front of 715? I always get the full path when I run a -ls command. I would expect either:

/715/file.csv
or
/user/<me>/715/file.csv

Or is that what you meant by "(didn't leave rest of ls results)"?

dean
On Fri, Feb 15, 2013 at 10:45 AM, Joseph D Antoni <[EMAIL PROTECTED]> wrote:

[cloudera@localhost data]$ hdfs dfs -ls 715
>Found 13 items
>-rw-r--r--   1 cloudera cloudera    7853975 2013-02-15 00:41 715/file.csv (didn't leave rest of ls results)
>
>
>Thanks on the directory--wasn't clear on that..
>
>Joey
>
>
>
>
>
>
>
>
>________________________________
> From: Dean Wampler <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]; Joseph D Antoni <[EMAIL PROTECTED]>
>Sent: Friday, February 15, 2013 11:37 AM
>Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories
>
>
>
>You confirmed that 715 is an actual directory? It didn't become a file by accident?
>
>
>By the way, you don't need to include the file name in the LOCATION. It will read all the files in the directory.
>
>
>dean
>
>
>On Fri, Feb 15, 2013 at 10:29 AM, Joseph D Antoni <[EMAIL PROTECTED]> wrote:
>
>I'm trying to create a series of external tables for a time series of data (using the prebuilt Cloudera VM).
>>
>>
>>The directory structure in HDFS is as such:
>>
>>
>>/711
>>/712
>>/713
>>/714
>>/715
>>/716
>>/717
>>
>>
>>Each directory contains the same set of files, from a different day. They were all put into HDFS using the following script:
>>
>>
>>for i in *;do hdfs dfs -put $i in $dir;done
>>
>>
>>They all show up with the same ownership/perms in HDFS.
>>
>>
>>Going into Hive to build the tables, I built a set of scripts to do the loads--then did a sed (changing 711 to 712,713, etc) to a file for each day. All of my loads work, EXCEPT for 715 and 716. 
>>
>>
>>Script is as follows:
>>
>>
>>create external table 715_table_name
>>(col1 string,
>>col2 string)
>>row format
>>delimited fields terminated by ','
>>lines terminated by '\n'
>>stored as textfile
>>location '/715/file.csv';
>>
>>
>>This is failing with:
>>
>>
>>Error in Metadata MetaException(message:Got except: org.apache.hadoop.fs.FileAlreadyExistsException Parent Path is not a directory: /715 715...
>>
>>
>>Like I mentioned it works for all of the other directories, except 715 and 716. Thoughts on troubleshooting path?
>>
>>
>>Thanks
>>
>>
>>Joey D'Antoni
>
>
>
>--
>Dean Wampler, Ph.D.
>thinkbiganalytics.com
>+1-312-339-1330
>
>
>
>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com
+1-312-339-1330

Hi Joseph

There are differences in the following ls commands

cloudera@localhost data]$ hdfs dfs -ls /715

This would list out all the contents in /715 in hdfs, if it is a dir

Found 1 items
-rw-r--r--   1 cloudera supergroup    7853975 2013-02-14 17:03 /715

The output clearly defines it is file as d is missing as the first char

[cloudera@localhost data]$ hdfs dfs -ls 715

This lists the dir 715 in your user's hdfs home dir. If your user is cloudera then usually your home dir might be /userdata/cloudera/ so in effect the dir listed is /userdata/cloudera/715
Regards
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: Joseph D Antoni <[EMAIL PROTECTED]>
Date: Fri, 15 Feb 2013 08:55:50
To: [EMAIL PROTECTED]<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories

Not sure--I just truncated the file list from the ls--that was the first file (just obfuscated the name)

The command I used to create the directories was:

hdfs dfs -mkdir 715
then
hdfs dfs -put myfile.csv 715

[cloudera@localhost data]$ hdfs dfs -ls /715
Found 1 items
-rw-r--r--   1 cloudera supergroup    7853975 2013-02-14 17:03 /715
[cloudera@localhost data]$ hdfs dfs -ls 715
Found 13 items
-rw-r--r--   1 cloudera cloudera    7853975 2013-02-15 00:41 715/40-file.csv

Thanks

________________________________
 From: Dean Wampler <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Joseph D Antoni <[EMAIL PROTECTED]>
Sent: Friday, February 15, 2013 11:50 AM
Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories
 

Something's odd about this output; why is there no / in front of 715? I always get the full path when I run a -ls command. I would expect either:

/715/file.csv
or
/user/<me>/715/file.csv

Or is that what you meant by "(didn't leave rest of ls results)"?

dean
On Fri, Feb 15, 2013 at 10:45 AM, Joseph D Antoni <[EMAIL PROTECTED]> wrote:

[cloudera@localhost data]$ hdfs dfs -ls 715
>Found 13 items
>-rw-r--r--   1 cloudera cloudera    7853975 2013-02-15 00:41 715/file.csv (didn't leave rest of ls results)
>
>
>Thanks on the directory--wasn't clear on that..
>
>Joey
>
>
>
>
>
>
>
>
>________________________________
> From: Dean Wampler <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]; Joseph D Antoni <[EMAIL PROTECTED]>
>Sent: Friday, February 15, 2013 11:37 AM
>Subject: Re: CREATE EXTERNAL TABLE Fails on Some Directories
>
>
>
>You confirmed that 715 is an actual directory? It didn't become a file by accident?
>
>
>By the way, you don't need to include the file name in the LOCATION. It will read all the files in the directory.
>
>
>dean
>
>
>On Fri, Feb 15, 2013 at 10:29 AM, Joseph D Antoni <[EMAIL PROTECTED]> wrote:
>
>I'm trying to create a series of external tables for a time series of data (using the prebuilt Cloudera VM).
>>
>>
>>The directory structure in HDFS is as such:
>>
>>
>>/711
>>/712
>>/713
>>/714
>>/715
>>/716
>>/717
>>
>>
>>Each directory contains the same set of files, from a different day. They were all put into HDFS using the following script:
>>
>>
>>for i in *;do hdfs dfs -put $i in $dir;done
>>
>>
>>They all show up with the same ownership/perms in HDFS.
>>
>>
>>Going into Hive to build the tables, I built a set of scripts to do the loads--then did a sed (changing 711 to 712,713, etc) to a file for each day. All of my loads work, EXCEPT for 715 and 716. 
>>
>>
>>Script is as follows:
>>
>>
>>create external table 715_table_name
>>(col1 string,
>>col2 string)
>>row format
>>delimited fields terminated by ','
>>lines terminated by '\n'
>>stored as textfile
>>location '/715/file.csv';
>>
>>
>>This is failing with:
>>
>>
>>Error in Metadata MetaException(message:Got except: org.apache.hadoop.fs.FileAlreadyExistsException Parent Path is not a directory: /715 715...
>>
>>
>>Like I mentioned it works for all of the other directories, except 715 and 716. Thoughts on troubleshooting path?
>>
>>
>>Thanks
Dean Wampler, Ph.D.
thinkbiganalytics.com
+1-312-339-1330