Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Migrating pseudo distribute map reduce with user jar file to 0.90.1


Copy link to this message
-
Re: Migrating pseudo distribute map reduce with user jar file to 0.90.1
With CDH3B4, the hadoop processes run as separate users (like hdfs,
mapred, etc). Did you set the CDH3B4 directory permissions correctly
as described in the install document?
See: https://ccp.cloudera.com/display/CDHDOC/Upgrading+to+CDH3 and
search for "permissions".
Also see this: https://ccp.cloudera.com/display/CDHDOC/Upgrading+to+CDH3#UpgradingtoCDH3-ChangesinUserAccountsandGroupsinCDH3DuetoSecurity

In this case, make sure that the /tmp/,,,/staging/... directory is
accessible by the task user account (most likely mapred). What seems
to be happenning in your case is that the task is not able to access
your /tmp directory due to user permissions.

If this still doesn't work - try running one of the MR
hadoop-*-examples (like "pi") so that you can validate that the
cluster is setup correctly.
--Suraj
On Mon, Apr 25, 2011 at 9:18 PM, Pete Tyler <[EMAIL PROTECTED]> wrote:
>
> My map reduce jobs, which were running fine with HBase 0.20.4 with Hadoop
> 0.20.2 are now failing as I try to upgrade to HBase 0.90.1 with Hadoop
> 0.20.2-CDH3B4.
>
> Under 0.90.1 I see the following error,
>
> Error initializing attempt_201104252111_0001_m_000002_0:
> java.io.FileNotFoundException: File
> /tmp/hadoop-peter/mapred/staging/peter/.staging/job_201104252111_0001/job.jar
> does not exist.
>        at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383)
>        at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
>        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
>        at
> org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:61)
>        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1303)
>        at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
>        at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
>        at
> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
>        at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:198)
>        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1154)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>        at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1129)
>        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1055)
>        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2212)
>        at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2176)
>
> My code, which worked fine under 0.20.4 looks like this,
>
>        Configuration conf = HBaseConfiguration.create();
>        conf.set("mapred.job.tracker", "localhost:9001");
>
>        Job job = new Job(conf);
>        JobConf jobConf = job.getConfiguration();
>        jobConf.setJar( dirName + "/" + jarName);
>
>        job.setJarByClass(MyMapper.class);
>        job.setJobName("My MapReduce Pseudo Distributed");
>
>        Scan scan = new Scan();
>
>        String tableNameIn  = "MyTableIn";
>        String tableNameOut = "MyTableOut";
>
>        TableMapReduceUtil.initTableMapperJob(
>                  tableNameIn,
>                  scan,
>                  MyMapper.class,
>                  ImmutableBytesWritable.class,
>                  ImmutableBytesWritable.class,
>                 job
>         );
>
>        TableMapReduceUtil.initTableReducerJob(
>               tableNameOut,
>               MyReducer.class,
>               job
>         );
>
>        job.setOutputFormatClass(TableOutputFormat.class);
>        job.setNumReduceTasks(1);
>
>        job.waitForCompletion(true);
>
> Any pointers on how to migrate my code so it works on 0.90.1 would be
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB