Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> cannot submit a job via java client in hadoop- 2.0.5-alpha


+
Francis.Hu 2013-07-10, 08:33
+
Azuryy Yu 2013-07-10, 10:51
Copy link to this message
-
RE: cannot submit a job via java client in hadoop- 2.0.5-alpha
Hi Francis,

                Could you check whether those configuration files are getting loaded or not, There could be a chance that these configuration files are not getting loaded into configuration object due to some invalid path reason.

        conf1.addResource("resources/mapred-site.xml");
       // Can you try printing the properties 'yarn.resourcemanager.address' & 'mapreduce.framework.name' values and check they are   coming as per the config files
        JobConf conf = new JobConf(conf1);

Thanks
Devaraj k

From: Azuryy Yu [mailto:[EMAIL PROTECTED]]
Sent: 10 July 2013 16:22
To: [EMAIL PROTECTED]
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha

you didn't set yarn.nodemanager.address in your yarn-site.xml

On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,All

I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is never submitted successfully. It throws out exception all the time.
My configurations are attached.  Can anyone help me? Thanks.

---------my-java client
public class TestJob {

    public void execute() {

        Configuration conf1 = new Configuration();
        conf1.addResource("resources/core-site.xml");
        conf1.addResource("resources/hdfs-site.xml");
        conf1.addResource("resources/yarn-site.xml");
        conf1.addResource("resources/mapred-site.xml");
        JobConf conf = new JobConf(conf1);

        conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
        conf.setJobName("Test");

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);

        conf.setMapperClass(DisplayRequestMapper.class);
        conf.setReducerClass(DisplayRequestReducer.class);

        FileInputFormat.setInputPaths(conf,new Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
        FileOutputFormat.setOutputPath(conf, new Path("/home/francis/hadoop-jobs/result/"));

        try {
            JobClient client = new JobClient(conf);
            RunningJob job = client.submitJob(conf);
            job.waitForCompletion();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

----------Exception

jvm 1    | java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name<http://mapreduce.framework.name> and the correspond server addresses.
jvm 1    |      at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1    |      at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1    |      at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1    |      at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1    |      at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1    |      at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
+
Devaraj k 2013-07-10, 11:21
+
hadoop hive 2013-07-10, 10:39
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB