Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - HDFS to S3 copy issues


Copy link to this message
-
Re: HDFS to S3 copy issues
Nitin Pawar 2012-07-06, 07:12
you may want to try following command

instead of using hdfs try hftp

hadoop -i -ppgu -log /tmp/mylog -m 20 distcp hftp://servername:port/path
 (hdfs://target.server:port/path | s3://id:sercret@domain)

On Fri, Jul 6, 2012 at 12:19 PM, Momina Khan <[EMAIL PROTECTED]> wrote:

> hi Ivan,
>
> i have tried with both ports 9000 and 9001 i get the same error dump ...
>
> best
> momina
>
> On Fri, Jul 6, 2012 at 11:01 AM, Ivan Mitic <[EMAIL PROTECTED]> wrote:
>
> > Hi Momina,
> >
> > Could it be that you misspelled the port in your source path, you mind
> > trying with: hdfs://10.240.113.162:9000/data/
> >
> > Ivan
> >
> > -----Original Message-----
> > From: Momina Khan [mailto:[EMAIL PROTECTED]]
> > Sent: Thursday, July 05, 2012 10:30 PM
> > To: [EMAIL PROTECTED]
> > Subject: HDFS to S3 copy issues
> >
> > hi ... hope someone is able to help me out with this ... have tried an
> > exhaustive search of google and AWS forum but there is little help in
> this
> > regard and all that i found didnt work for me!
> >
> > i want to copy data from HDFS to my S3 bucket ... to test whether my HDFS
> > url is correct i tried the fs -cat command which works just fine ...
> spits
> > contents of the file ubuntu@domU-12-31-39-04-6E-58
> :/state/partition1/hadoop-1.0.1$
> > *bin/hadoop fs -cat hdfs://10.240.113.162:9000/data/hello.txt*
> >
> > but when i try to distance copy the file from hdfs (same location as
> > above) to my s3 bucket it says connection to server refused! have looked
> up
> > Google exhaustively but cannot get an answer. they say that the port may
> be
> > blocked but have checked that 9000-9001 are not blocked .... could it be
> an
> > autghentication issue? just saying ... out of ideas.
> >
> > Find the call trace attached below:
> >
> > ubuntu@domU-12-31-39-04-6E-58:/state/partition1/hadoop-1.0.1$
> *bin/hadoop
> > distcp hdfs://10.240.113.162:9001/data/ s3://ID:**SECRET@momina
> > *
> >
> > 12/07/05 12:48:37 INFO tools.DistCp: srcPaths=[hdfs://
> > 10.240.113.162:9001/data]
> > 12/07/05 12:48:37 INFO tools.DistCp: destPath=s3://ID:SECRET@momina
> >
> > 12/07/05 12:48:38 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 0 time(s).
> > 12/07/05 12:48:39 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 1 time(s).
> > 12/07/05 12:48:40 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 2 time(s).
> > 12/07/05 12:48:41 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 3 time(s).
> > 12/07/05 12:48:42 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 4 time(s).
> > 12/07/05 12:48:43 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 5 time(s).
> > 12/07/05 12:48:44 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 6 time(s).
> > 12/07/05 12:48:45 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 7 time(s).
> > 12/07/05 12:48:46 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 8 time(s).
> > 12/07/05 12:48:47 INFO ipc.Client: Retrying connect to server:
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
> > tried 9 time(s).
> > With failures, global counters are inaccurate; consider running with -i
> > Copy failed: java.net.ConnectException: Call to
> > domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001 failed on
> > connection exception: java.net.ConnectException: Connection refused

Nitin Pawar