Hi,

I'd like to move and copy files from one directory in HDFS to another
one. I know there are methods in the Filesystem API that enable
copying files between the local disk and HDFS, but I couldn't figure
out how to do this between two paths both in HDFS. I think rename(Path
src, Path dest) can be used to move files, but copying still remains a
challenge to me. Any ideas?

Thanks,
Jim

There is no FileSystem API to copy.
You could try
hadoop dfs -cp <src> <dest>

which basically reads the file and writes to new file.
The code for this is in FsShell.java
----- Original Message ----
From: Jim Twensky <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Fri, May 13, 2011 1:21:09 PM
Subject: FileSystem API - Moving files in HDFS

Hi,

I'd like to move and copy files from one directory in HDFS to another
one. I know there are methods in the Filesystem API that enable
copying files between the local disk and HDFS, but I couldn't figure
out how to do this between two paths both in HDFS. I think rename(Path
src, Path dest) can be used to move files, but copying still remains a
challenge to me. Any ideas?

Thanks,
Jim

Jim,
 you can use FileUtil.copy() methods to copy files.

Hope that helps.
--
thanks
mahadev
@mahadevkonar

On Fri, May 13, 2011 at 2:00 PM, lohit <[EMAIL PROTECTED]> wrote:
> There is no FileSystem API to copy.
> You could try
> hadoop dfs -cp <src> <dest>
>
> which basically reads the file and writes to new file.
> The code for this is in FsShell.java
>
>
> ----- Original Message ----
> From: Jim Twensky <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Fri, May 13, 2011 1:21:09 PM
> Subject: FileSystem API - Moving files in HDFS
>
> Hi,
>
> I'd like to move and copy files from one directory in HDFS to another
> one. I know there are methods in the Filesystem API that enable
> copying files between the local disk and HDFS, but I couldn't figure
> out how to do this between two paths both in HDFS. I think rename(Path
> src, Path dest) can be used to move files, but copying still remains a
> challenge to me. Any ideas?
>
> Thanks,
> Jim
>
>

If they are lots of large files, and you need to copy them quickly, i.e. Not have all the data go through a single machine, you can use hadoop distcp too.

--Bobby

On 5/14/11 12:49 AM, "Mahadev Konar" <[EMAIL PROTECTED]> wrote:

Jim,
 you can use FileUtil.copy() methods to copy files.

Hope that helps.
--
thanks
mahadev
@mahadevkonar

On Fri, May 13, 2011 at 2:00 PM, lohit <[EMAIL PROTECTED]> wrote:
> There is no FileSystem API to copy.
> You could try
> hadoop dfs -cp <src> <dest>
>
> which basically reads the file and writes to new file.
> The code for this is in FsShell.java
>
>
> ----- Original Message ----
> From: Jim Twensky <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Fri, May 13, 2011 1:21:09 PM
> Subject: FileSystem API - Moving files in HDFS
>
> Hi,
>
> I'd like to move and copy files from one directory in HDFS to another
> one. I know there are methods in the Filesystem API that enable
> copying files between the local disk and HDFS, but I couldn't figure
> out how to do this between two paths both in HDFS. I think rename(Path
> src, Path dest) can be used to move files, but copying still remains a
> challenge to me. Any ideas?
>
> Thanks,
> Jim
>
>