Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - rename operation


Copy link to this message
-
rename operation
Jean-Pierre OCALAN 2010-08-25, 00:21
Hi,

I would like to know if the *rename* operation (i.e. renaming a directory or
a single file) can be consider as an atomic operation in HDFS.

Basically what i am trying to achieve is having one process that
continiously add new file into the HDFS and another process that will start
every 15 minutes a map/reduce flow on file that were newly added into the
HDFS.

In other words a process A continuously read a *local directory
"A/in"*where new files are moved there continuously and put each file
in a
*"A/tmp" directory on the HDFS*. When A finish to put one file in
"*A/tmp"*it will
*move/rename that file into a "B/in" directory*. At the same time a process
B will, every 15 minutes, push all the files present in "B/in" to a
map/reduce flow.

Regards,

-- JP