Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> cannot submit a job via java client in hadoop- 2.0.5-alpha


+
Francis.Hu 2013-07-10, 08:33
+
Azuryy Yu 2013-07-10, 10:51
Copy link to this message
-
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Hi Francis,

                Could you check whether those configuration files are getting loaded or not, There could be a chance that these configuration files are not getting loaded into configuration object due to some invalid path reason.

        conf1.addResource("resources/mapred-site.xml");
       // Can you try printing the properties 'yarn.resourcemanager.address' & 'mapreduce.framework.name' values and check they are   coming as per the config files
        JobConf conf = new JobConf(conf1);

Thanks
Devaraj k

From: Azuryy Yu [mailto:[EMAIL PROTECTED]]
Sent: 10 July 2013 16:22
To: [EMAIL PROTECTED]
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha

you didn't set yarn.nodemanager.address in your yarn-site.xml

On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,All

I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached.  Can anyone help me? Thanks.

---------my-java client
public class TestJob {

    public void execute() {

        Configuration conf1 = new Configuration();
        conf1.addResource("resources/core-site.xml");
        conf1.addResource("resources/hdfs-site.xml");
        conf1.addResource("resources/yarn-site.xml");
        conf1.addResource("resources/mapred-site.xml");
        JobConf conf = new JobConf(conf1);

        conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
        conf.setJobName("Test");

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);

        conf.setMapperClass(DisplayRequestMapper.class);
        conf.setReducerClass(DisplayRequestReducer.class);

        FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
        FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));

        try {
            JobClient client = new JobClient(conf);
            RunningJob job = client.submitJob(conf);
            job.waitForCompletion();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

----------Exception

jvm 1    | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1    |      at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1    |      at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1    |      at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1    |      at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1    |      at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1    |      at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
+
Devaraj k 2013-07-10, 11:21
+
hadoop hive 2013-07-10, 10:39