Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - transfer large file onto hdfs


+
Ted Yu 2012-07-05, 18:24
+
Harsh J 2012-07-06, 06:47
Copy link to this message
-
Re: transfer large file onto hdfs
Ted Yu 2012-07-06, 13:58
Thanks for the reply, Harsh.

BTW Does anyone deploy cdh3u4 on Solaris ?

On Thu, Jul 5, 2012 at 11:47 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Should we assume that the non-hadoop system has no way to get on the
> network of the hadoop cluster and its clients? Otherwise, you can
> grant it temporary access and do a write from it itself. If not:
>
> Run a micro FTP server pointed to that file, and then do a 'hadoop fs
> -cp ftp://location hdfs://location', since FTPFileSystem is present in
> Hadoop? Or if NFS/etc. mounted, file:/// will work (or via
> copyFromLocal/put). Essentially you're bringing in the file remotely
> but performing the copy via CLI.
>
> Or you can copy them in chunks, either keeping the destination file
> writer open, if possible, or appending (depending on what version of
> Hadoop you're using).
>
> On Thu, Jul 5, 2012 at 11:54 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Hi,
> > One of the customers wants to transfer dump file (size ~ 2TB) from
> outside
> > hadoop cluster onto hdfs.
> > The size exceeds free space on CLI machine.
> >
> > I want to poll best practice in this scenario.
> >
> > Thanks
>
>
>
> --
> Harsh J
>
+
Harsh J 2012-07-06, 14:55