Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: does "fs -put " create subdirectories?


Copy link to this message
-
RE: does "fs -put " create subdirectories?
Modifying the script to precede the put with a mkdir (which creates the missing subdirectories) fixes the issue.thanksJohn

Date: Wed, 16 Jan 2013 12:36:20 -0500
Subject: Re: does "fs -put " create subdirectories?
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]

Good point Harsh.  As a Linux Admin, I prefer the behavior of 2.x.  It allows me to see if I've made a mistake in my planned placement of files instead of blindly writing.

On Wed, Jan 16, 2013 at 12:05 PM, Harsh J <[EMAIL PROTECTED]> wrote:

On 1.x, -put does mkdir the parent directories if they are non existent (akin to mkdir -p).

On 2.x, -put does not do this - we need the dir to pre-exist or mkdir'd before -put.
➜  ~  hadoop fs -put foo a/bc/dfe/fsdf.tput: `a/bc/dfe/fsdf.t': No such file or directory
On Wed, Jan 16, 2013 at 10:28 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]> wrote:
Yes it does, you can just try ;)

hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls /user/

Found 1 items

drwxr-xr-x   - hbase supergroup          0 2013-01-03 09:54 /user/hbase

hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -put CHANGES.txt

/user/test/CHANGES.txt

hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls /user/

Found 2 items

drwxr-xr-x   - hbase  supergroup          0 2013-01-03 09:54 /user/hbase

drwxr-xr-x   - hadoop supergroup          0 2013-01-16 11:56 /user/test

hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls /user/test/

Found 1 items

-rw-r--r--   3 hadoop supergroup     446615 2013-01-16 11:56

/user/test/CHANGES.txt

hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -put CHANGES.txt

/user/test2/subdir/anotherone/CHANGES.txt

hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls /user/test2/subdir/anotherone/

Found 1 items

-rw-r--r--   3 hadoop supergroup     446615 2013-01-16 11:58

/user/test2/subdir/anotherone/CHANGES.txt

2013/1/16, John Meza <[EMAIL PROTECTED]>:

> Does "hadoop fs -put mmddyyyy.tsv t1/2012/01/01/mmddyyyy.tsv" create the

> necassary subdirectories in hdfs?thanksJohn

--
Harsh J
     
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB