Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - 答复: cannot submit a job via java client in hadoop- 2.0.5-alpha


Copy link to this message
-
答复: cannot submit a job via java client in hadoop- 2.0.5-alpha
Francis.Hu 2013-07-11, 02:46
Hi, Devaraj k  and Azuryy Yu

 

Thanks both of you.

I just get it resolved.  The problem is that below highlighted jar is not
included in my java client side so that when the Job is initializing, it can
not find the class YarnClientProtocolProvider to do further initialization.
Then it causes the exception in Cluster.initialize() is thrown out.

hadoop-mapreduce-client-jobclient-2.0.5-alpha.jar

 

Thanks,

Francis.Hu

 

 

发件人: Devaraj k [mailto:[EMAIL PROTECTED]]
发送时间: Wednesday, July 10, 2013 19:11
收件人: [EMAIL PROTECTED]
主题: RE: cannot submit a job via java client in hadoop- 2.0.5-alpha

 

Hi Francis,

 

                Could you check whether those configuration files are
getting loaded or not, There could be a chance that these configuration
files are not getting loaded into configuration object due to some invalid
path reason.

 

        conf1.addResource("resources/mapred-site.xml");

       // Can you try printing the properties
‘yarn.resourcemanager.address’ & ‘mapreduce.framework.name’ values and
check they are   coming as per the config files    

        JobConf conf = new JobConf(conf1);

 

    

 

Thanks

Devaraj k

 

From: Azuryy Yu [mailto:[EMAIL PROTECTED]]
Sent: 10 July 2013 16:22
To: [EMAIL PROTECTED]
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha

 

you didn't set yarn.nodemanager.address in your yarn-site.xml

 

 

 

On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu <[EMAIL PROTECTED]>
wrote:

Hi,All

 

I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource
Manager.

I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.

My configurations are attached.  Can anyone help me? Thanks.

 

---------my-java client

public class TestJob {

    

    public void execute() {

 

        Configuration conf1 = new Configuration();

        conf1.addResource("resources/core-site.xml");

        conf1.addResource("resources/hdfs-site.xml");

        conf1.addResource("resources/yarn-site.xml");

        conf1.addResource("resources/mapred-site.xml");

        JobConf conf = new JobConf(conf1);

        

        conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");

        conf.setJobName("Test");

 

        conf.setInputFormat(TextInputFormat.class);

        conf.setOutputFormat(TextOutputFormat.class);

 

        conf.setOutputKeyClass(Text.class);

        conf.setOutputValueClass(IntWritable.class);

 

        conf.setMapperClass(DisplayRequestMapper.class);

        conf.setReducerClass(DisplayRequestReducer.class);

 

        FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));

        FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));

 

        try {

            JobClient client = new JobClient(conf);

            RunningJob job = client.submitJob(conf);

            job.waitForCompletion();

        } catch (IOException e) {

            e.printStackTrace();

        }

    }

}

 

----------Exception

 

jvm 1    | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.

jvm 1    |      at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)

jvm 1    |      at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)

jvm 1    |      at
org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)

jvm 1    |      at
org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)

jvm 1    |      at
org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)

jvm 1    |      at
com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)

 

 

Thanks,

Francis.Hu