Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Migrating pseudo distribute map reduce with user jar file to 0.90.1


Copy link to this message
-
Re: Migrating pseudo distribute map reduce with user jar file to 0.90.1
Suraj Varma 2011-04-26, 05:02
With CDH3B4, the hadoop processes run as separate users (like hdfs,
mapred, etc). Did you set the CDH3B4 directory permissions correctly
as described in the install document?
See: https://ccp.cloudera.com/display/CDHDOC/Upgrading+to+CDH3 and
search for "permissions".
Also see this: https://ccp.cloudera.com/display/CDHDOC/Upgrading+to+CDH3#UpgradingtoCDH3-ChangesinUserAccountsandGroupsinCDH3DuetoSecurity

In this case, make sure that the /tmp/,,,/staging/... directory is
accessible by the task user account (most likely mapred). What seems
to be happenning in your case is that the task is not able to access
your /tmp directory due to user permissions.

If this still doesn't work - try running one of the MR
hadoop-*-examples (like "pi") so that you can validate that the
cluster is setup correctly.
--Suraj
On Mon, Apr 25, 2011 at 9:18 PM, Pete Tyler <[EMAIL PROTECTED]> wrote:
>
> My map reduce jobs, which were running fine with HBase 0.20.4 with Hadoop
> 0.20.2 are now failing as I try to upgrade to HBase 0.90.1 with Hadoop
> 0.20.2-CDH3B4.
>
> Under 0.90.1 I see the following error,
>
> Error initializing attempt_201104252111_0001_m_000002_0:
> java.io.FileNotFoundException: File
> /tmp/hadoop-peter/mapred/staging/peter/.staging/job_201104252111_0001/job.jar
> does not exist.
>        at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383)
>        at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
>        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
>        at
> org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:61)
>        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1303)
>        at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
>        at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
>        at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
>        at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:198)
>        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1154)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>        at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1129)
>        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1055)
>        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2212)
>        at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2176)
>
> My code, which worked fine under 0.20.4 looks like this,
>
>        Configuration conf = HBaseConfiguration.create();
>        conf.set("mapred.job.tracker", "localhost:9001");
>
>        Job job = new Job(conf);
>        JobConf jobConf = job.getConfiguration();
>        jobConf.setJar( dirName + "/" + jarName);
>
>        job.setJarByClass(MyMapper.class);
>        job.setJobName("My MapReduce Pseudo Distributed");
>
>        Scan scan = new Scan();
>
>        String tableNameIn  = "MyTableIn";
>        String tableNameOut = "MyTableOut";
>
>        TableMapReduceUtil.initTableMapperJob(
>                  tableNameIn,
>                  scan,
>                  MyMapper.class,
>                  ImmutableBytesWritable.class,
>                  ImmutableBytesWritable.class,
>                 job
>         );
>
>        TableMapReduceUtil.initTableReducerJob(
>               tableNameOut,
>               MyReducer.class,
>               job
>         );
>
>        job.setOutputFormatClass(TableOutputFormat.class);
>        job.setNumReduceTasks(1);
>
>        job.waitForCompletion(true);
>
> Any pointers on how to migrate my code so it works on 0.90.1 would be