Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: copy files from ftp to hdfs in parallel, distcp failed


Copy link to this message
-
Re: copy files from ftp to hdfs in parallel, distcp failed
Ram 2013-07-16, 15:03
Hi,
    Please replace 0.0.0.0.with your ftp host ip address and try it.

Hi,

From,
Ramesh.
On Mon, Jul 15, 2013 at 3:22 PM, Hao Ren <[EMAIL PROTECTED]> wrote:

>  Thank you, Ram
>
> I have configured core-site.xml as following:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>
>     <property>
>         <name>hadoop.tmp.dir</name>
>         <value>/vol/persistent-hdfs</value>
>     </property>
>
>     <property>
>         <name>fs.default.name</name>
>         <value>hdfs://ec2-23-23-33-234.compute-1.amazonaws.com:9010
> </value>
>     </property>
>
>     <property>
>         <name>io.file.buffer.size</name>
>         <value>65536</value>
>     </property>
>
>     <property>
>         <name>fs.ftp.host</name>
>         <value>0.0.0.0</value>
>     </property>
>
>     <property>
>         <name>fs.ftp.host.port</name>
>         <value>21</value>
>     </property>
>
> </configuration>
>
> Then I tried  hadoop fs -ls file:/// , it works.
> But hadoop fs -ls ftp://<login>:<password>@<ftp server ip>/<directory>/
> doesn't work as usual:
>     ls: Cannot access ftp://<user>:<password>@<ftp server
> ip>/<directory>/: No such file or directory.
>
> When ignoring <directroy> as :
>
> hadoop fs -ls ftp://<login>:<password>@<ftp server ip>/
>
> There are no error msgs, but it lists nothing.
>
>
> I have also check the rights for my /home/<user> directroy:
>
> drwxr-xr-x 11 <user> <user>  4096 jui 11 16:30 <user>
>
> and all the files under /home/<user> have rights 755.
>
> I can easily copy the link ftp://<user>:<password>@<ftp server
> ip>/<directory>/ to firefox, it lists all the files as expected.
>
> Any workaround here ?
>
> Thank you.
>
> Le 12/07/2013 14:01, Ram a écrit :
>
> Please configure the following in core-ste.xml and try.
>    Use hadoop fs -ls file:///  -- to display local file system files
>    Use hadoop fs -ls ftp://<your ftp location>   -- to display ftp files
> if it is listing files go for distcp.
>
>  reference from
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml
>
>
>    fs.ftp.host 0.0.0.0 FTP filesystem connects to this server
> fs.ftp.host.port 21 FTP filesystem connects to fs.ftp.host on this port
>
>
>
> --
> Hao Ren
> ClaraVistawww.claravista.fr
>
>