Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> HDFS to S3 copy issues


Copy link to this message
-
Re: HDFS to S3 copy issues
you may want to try following command

instead of using hdfs try hftp

hadoop -i -ppgu -log /tmp/mylog -m 20 distcp hftp://servername:port/path
 (hdfs://target.server:port/path | s3://id:sercret@domain)

On Fri, Jul 6, 2012 at 12:19 PM, Momina Khan <[EMAIL PROTECTED]> wrote:

> hi Ivan,
>
> i have tried with both ports 9000 and 9001 i get the same error dump ...
>
> best
> momina
>
> On Fri, Jul 6, 2012 at 11:01 AM, Ivan Mitic <[EMAIL PROTECTED]> wrote:
>
> > Hi Momina,
> >
> > Could it be that you misspelled the port in your source path, you mind
> > trying with: hdfs://10.240.113.162:9000/data/
> >
> > Ivan
> >
> > -----Original Message-----
> > From: Momina Khan [mailto:[EMAIL PROTECTED]]
> > Sent: Thursday, July 05, 2012 10:30 PM
> > To: [EMAIL PROTECTED]
> > Subject: HDFS to S3 copy issues
> >
> > hi ... hope someone is able to help me out with this ... have tried an
> > exhaustive search of google and AWS forum but there is little help in
> this
> > regard and all that i found didnt work for me!
> >
> > i want to copy data from HDFS to my S3 bucket ... to test whether my HDFS
> > url is correct i tried the fs -cat command which works just fine ...
> spits
> > contents of the file ubuntu@domU-12-31-39-04-6E-58
> :/state/partition1/hadoop-1.0.1$
> > *bin/hadoop fs -cat hdfs://10.240.113.162:9000/data/hello.txt*
> >
> > but when i try to distance copy the file from hdfs (same location as
> > above) to my s3 bucket it says connection to server refused! have looked
> up
> > Google exhaustively but cannot get an answer. they say that the port may
> be
> > blocked but have checked that 9000-9001 are not blocked .... could it be
> an
> > autghentication issue? just saying ... out of ideas.
> >
> > Find the call trace attached below:
> >
> > ubuntu@domU-12-31-39-04-6E-58:/state/partition1/hadoop-1.0.1$
> *bin/hadoop
> > distcp hdfs://10.240.113.162:9001/data/ s3://ID:**SECRET@momina
> > *
> >
> > 12/07/05 12:48:37 INFO tools.DistCp: srcPaths=[hdfs://
> > 10.240.113.162:9001/data]
> > 12/07/05 12:48:37 INFO tools.DistCp: destPath=s3://ID:SECRET@momina
> >
> > 12/07/05 12:48:38 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 0 time(s).
> > 12/07/05 12:48:39 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 1 time(s).
> > 12/07/05 12:48:40 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 2 time(s).
> > 12/07/05 12:48:41 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 3 time(s).
> > 12/07/05 12:48:42 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 4 time(s).
> > 12/07/05 12:48:43 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 5 time(s).
> > 12/07/05 12:48:44 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 6 time(s).
> > 12/07/05 12:48:45 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 7 time(s).
> > 12/07/05 12:48:46 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 8 time(s).
> > 12/07/05 12:48:47 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 9 time(s).
> > With failures, global counters are inaccurate; consider running with -i
> > Copy failed: java.net.ConnectException: Call to
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001 failed on
> > connection exception: java.net.ConnectException: Connection refused

Nitin Pawar
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB