Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: problem using s3 instead of hdfs


Copy link to this message
-
Re: problem using s3 instead of hdfs
Hi,

I've not tried this on S3. However, the directory mentioned in the
exception is based on the value of this particular configuration
key: mapreduce.jobtracker.staging.root.dir. This defaults
to ${hadoop.tmp.dir}/mapred/staging. Can you please set this to an S3
location and try ?

Thanks
Hemanth

On Mon, Oct 15, 2012 at 10:43 PM, Parth Savani <[EMAIL PROTECTED]>wrote:

> Hello,
>       I am trying to run hadoop on s3 using distributed mode. However I am
> having issues running my job successfully on it. I get the following error
> I followed the instructions provided in this article ->
> http://wiki.apache.org/hadoop/AmazonS3
> I replaced the fs.default.name value in my hdfs-site.xml to
> s3n://ID:SECRET@BUCKET
> And I am running my job using the following: hadoop jar
> /path/to/my/jar/abcd.jar /input /output
> Where */input* is the folder name inside the s3 bucket
> (s3n://ID:SECRET@BUCKET/input)
> and */output *folder should created in my bucket (s3n://ID:SECRET@BUCKET
> /output)
> Below is the error i get. It is looking for job.jar on s3 and that path is
> on my server from where i am launching my job.
>
> java.io.FileNotFoundException: No such file or directory
> '/opt/data/hadoop/hadoop-mapred/mapred/staging/psavani/.staging/job_201207021606_1036/job.jar'
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:412)
>  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
>  at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1371)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1352)
>  at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
> at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
>  at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:222)
>  at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1372)
> at java.security.AccessController.doPri
>
>
>
>
>