Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: copy files from ftp to hdfs in parallel, distcp failed


+
Balaji Narayanan 2013-07-11, 18:47
+
Hao Ren 2013-07-12, 07:34
+
Ram 2013-07-12, 12:01
Copy link to this message
-
Re: copy files from ftp to hdfs in parallel, distcp failed
Hi,
    Please replace 0.0.0.0.with your ftp host ip address and try it.

Hi,

From,
Ramesh.
On Mon, Jul 15, 2013 at 3:22 PM, Hao Ren <[EMAIL PROTECTED]> wrote:

>  Thank you, Ram
>
> I have configured core-site.xml as following:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>
>     <property>
>         <name>hadoop.tmp.dir</name>
>         <value>/vol/persistent-hdfs</value>
>     </property>
>
>     <property>
>         <name>fs.default.name</name>
>         <value>hdfs://ec2-23-23-33-234.compute-1.amazonaws.com:9010
> </value>
>     </property>
>
>     <property>
>         <name>io.file.buffer.size</name>
>         <value>65536</value>
>     </property>
>
>     <property>
>         <name>fs.ftp.host</name>
>         <value>0.0.0.0</value>
>     </property>
>
>     <property>
>         <name>fs.ftp.host.port</name>
>         <value>21</value>
>     </property>
>
> </configuration>
>
> Then I tried  hadoop fs -ls file:/// , it works.
> But hadoop fs -ls ftp://<login>:<password>@<ftp server ip>/<directory>/
> doesn't work as usual:
>     ls: Cannot access ftp://<user>:<password>@<ftp server
> ip>/<directory>/: No such file or directory.
>
> When ignoring <directroy> as :
>
> hadoop fs -ls ftp://<login>:<password>@<ftp server ip>/
>
> There are no error msgs, but it lists nothing.
>
>
> I have also check the rights for my /home/<user> directroy:
>
> drwxr-xr-x 11 <user> <user>  4096 jui 11 16:30 <user>
>
> and all the files under /home/<user> have rights 755.
>
> I can easily copy the link ftp://<user>:<password>@<ftp server
> ip>/<directory>/ to firefox, it lists all the files as expected.
>
> Any workaround here ?
>
> Thank you.
>
> Le 12/07/2013 14:01, Ram a écrit :
>
> Please configure the following in core-ste.xml and try.
>    Use hadoop fs -ls file:///  -- to display local file system files
>    Use hadoop fs -ls ftp://<your ftp location>   -- to display ftp files
> if it is listing files go for distcp.
>
>  reference from
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml
>
>
>    fs.ftp.host 0.0.0.0 FTP filesystem connects to this server
> fs.ftp.host.port 21 FTP filesystem connects to fs.ftp.host on this port
>
>
>
> --
> Hao Ren
> ClaraVistawww.claravista.fr
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB