MapReduce, mail # user - Re: Erro running pi programm

Erro running pi programm

Kartashov, Andy 2012-11-09, 17:36

Yinghua,

What mode are you running your hadoop in: Local/Pseud/Fully...?

Your hostname is not recognised

Your configuration setting seems to be wrong.

Hi, all

Could some help looking at this problem? I am setting up a four node cluster on EC2 and seems that the cluster is set up fine until I start testing.

I have tried password-less ssh from each node to all the nodes and there is no problem connecting. Any advice will be greatly appreciated!

[hduser@ip-XX-XX-XXX-XXX hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.4.jar pi -Dmapreduce.clientfactory.class.name<http://Dmapreduce.clientfactory.class.name>=org.apache.hadoop.mapred.YarnClientFactory -libjars share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.4.jar 16 10000
Number of Maps = 16
Samples per Map = 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Wrote input for Map #12
Wrote input for Map #13
Wrote input for Map #14
Wrote input for Map #15
Starting Job
12/11/09 12:02:59 INFO input.FileInputFormat: Total input paths to process : 16
12/11/09 12:02:59 INFO mapreduce.JobSubmitter: number of splits:16
12/11/09 12:02:59 WARN conf.Configuration: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
12/11/09 12:02:59 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
12/11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
12/11/09 12:02:59 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.job.name<http://mapred.job.name> is deprecated. Instead, use mapreduce.job.name<http://mapreduce.job.name>
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
12/11/09 12:03:00 INFO mapred.ResourceMgrDelegate: Submitted application application_1352478937343_0002 to ResourceManager at master/10.12.181.233:60400<http://10.12.181.233:60400>
12/11/09 12:03:00 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1352478937343_0002/
12/11/09 12:03:00 INFO mapreduce.Job: Running job: job_1352478937343_0002
12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 running in uber mode : false
12/11/09 12:03:01 INFO mapreduce.Job: map 0% reduce 0%
12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 failed with state FAILED due to: Application application_1352478937343_0002 failed 1 times due to Error launching appattempt_1352478937343_0002_000001. Got exception: java.lang.reflect.UndeclaredThrowableException
   at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
   at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:115)
   at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:258)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:722)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Yinghua java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost
   at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
   at $Proxy24.startContainer(Unknown Source)
   at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
   ... 5 more
Caused by: java.net.UnknownHostException: Yinghua For more details see: http://wiki.apache.org/hadoop/UnknownHost
   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
   at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:236)
   at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188)

Re: Erro running pi programm

yinghua hu 2012-11-09, 17:51

Hi, Andy

Thanks for suggestions!

I am running it on a four node cluster on EC2. All the services started
fine, Namenode, Datanode, ResourceManager, NodeManager and
JobHistoryServer. Each node can ssh to all the nodes without problem.

But problem appears when trying to run any job.
On Fri, Nov 9, 2012 at 9:36 AM, Kartashov, Andy <[EMAIL PROTECTED]>wrote:

> Yinghua,
>
>
>
> What mode are you running your hadoop in: Local/Pseud/Fully...?
>
>
>
> Your hostname is not recognised
>
>
>
> Your configuration setting seems to be wrong.
>
>
>
>
>
>
>
>
>
>
>
> Hi, all
>
>
>
> Could some help looking at this problem? I am setting up a four node
> cluster on EC2 and seems that the cluster is set up fine until I start
> testing.
>
>
>
> I have tried password-less ssh from each node to all the nodes and there
> is no problem connecting. Any advice will be greatly appreciated!
>
>
>
> [hduser@ip-XX-XX-XXX-XXX hadoop]$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.4.jar pi -
> Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory
> -libjars
> share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.4.jar 16 10000
>
> Number of Maps = 16
>
> Samples per Map = 10000
>
> Wrote input for Map #0
>
> Wrote input for Map #1
>
> Wrote input for Map #2
>
> Wrote input for Map #3
>
> Wrote input for Map #4
>
> Wrote input for Map #5
>
> Wrote input for Map #6
>
> Wrote input for Map #7
>
> Wrote input for Map #8
>
> Wrote input for Map #9
>
> Wrote input for Map #10
>
> Wrote input for Map #11
>
> Wrote input for Map #12
>
> Wrote input for Map #13
>
> Wrote input for Map #14
>
> Wrote input for Map #15
>
> Starting Job
>
> 12/11/09 12:02:59 INFO input.FileInputFormat: Total input paths to process
> : 16
>
> 12/11/09 12:02:59 INFO mapreduce.JobSubmitter: number of splits:16
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.job.classpath.files is
> deprecated. Instead, use mapreduce.job.classpath.files
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files is
> deprecated. Instead, use mapreduce.job.cache.files
>
> 12/11/09 12:02:59 WARN conf.Configuration:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.output.value.class is
> deprecated. Instead, use mapreduce.job.output.value.class
>
> 12/11/09 12:02:59 WARN conf.Configuration:
> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.reduce.speculative
>
> 12/11/09 12:02:59 WARN conf.Configuration:
> mapred.used.genericoptionsparser is deprecated. Instead, use
> mapreduce.client.genericoptionsparser.used
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.job.name is deprecated.
> Instead, use mapreduce.job.name
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.reduce.class is
> deprecated. Instead, use mapreduce.job.reduce.class
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.inputformat.class is
> deprecated. Instead, use mapreduce.job.inputformat.class
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.input.dir is deprecated.
> Instead, use mapreduce.input.fileinputformat.inputdir
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.outputformat.class is
> deprecated. Instead, use mapreduce.job.outputformat.class
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
>
> 12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files.timestamps

Regards,

Yinghua

Re: Erro running pi programm

yinghua hu 2012-11-09, 18:20

Here are my configuration files

core-site.xml

<configuration>
   <property>
   <name>fs.default.name</name>
   <value>hdfs://master:9000</value>
   </property>
   <property>

   <name>hadoop.tmp.dir</name>

   <value>/usr/local/hadoop/tmp</value>

   </property>

</configuration>
mapred-site.xml

<configuration>
   <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
   </property>

</configuration>

hdfs-site.xml

<configuration>
   <property>
   <name>dfs.replication</name>
   <value>2</value>
   </property>
   <property>

   <name>dfs.permissions</name>

   <value>false</value>

   </property>

   <property>

   <name>dfs.namenode.name.dir</name>

   <value>file:/home/hduser/yarn_data/hdfs/namenode</value>

   </property>

   <property>

   <name>dfs.datanode.data.dir</name>

   <value>file:/home/hduser/yarn_data/hdfs/datanode</value>

   </property>

</configuration>

yarn-site.xml

<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>

  <value>mapreduce.shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
   <name>yarn.nodemanager.log-aggregation-enable</name>
   <value>true</value>
</property>
<property>
   <name>yarn.resourcemanager.resource-tracker.address</name>
   <value>master:8050</value>
</property>
<property>
   <name>yarn.resourcemanager.scheduler.address</name>
   <value>master:8030</value>
</property>
<property>
   <name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

<property>

   <name>yarn.resourcemanager.address</name>

   <value>master:60400</value>

</property>

</configuration>

On Fri, Nov 9, 2012 at 9:51 AM, yinghua hu <[EMAIL PROTECTED]> wrote:

> Hi, Andy
>
> Thanks for suggestions!
>
> I am running it on a four node cluster on EC2. All the services started
> fine, Namenode, Datanode, ResourceManager, NodeManager and
> JobHistoryServer. Each node can ssh to all the nodes without problem.
>
> But problem appears when trying to run any job.
>
>
>
>
> On Fri, Nov 9, 2012 at 9:36 AM, Kartashov, Andy <[EMAIL PROTECTED]>wrote:
>
>> Yinghua,
>>
>>
>>
>> What mode are you running your hadoop in: Local/Pseud/Fully...?
>>
>>
>>
>> Your hostname is not recognised
>>
>>
>>
>> Your configuration setting seems to be wrong.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Hi, all
>>
>>
>>
>> Could some help looking at this problem? I am setting up a four node
>> cluster on EC2 and seems that the cluster is set up fine until I start
>> testing.
>>
>>
>>
>> I have tried password-less ssh from each node to all the nodes and there
>> is no problem connecting. Any advice will be greatly appreciated!
>>
>>
>>
>> [hduser@ip-XX-XX-XXX-XXX hadoop]$ bin/hadoop jar
>> share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.4.jar pi -
>> Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory
>> -libjars
>> share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.4.jar 16 10000
>>
>> Number of Maps = 16
>>
>> Samples per Map = 10000
>>
>> Wrote input for Map #0
>>
>> Wrote input for Map #1
>>
>> Wrote input for Map #2
>>
>> Wrote input for Map #3
>>
>> Wrote input for Map #4
>>
>> Wrote input for Map #5
>>
>> Wrote input for Map #6
>>
>> Wrote input for Map #7
>>
>> Wrote input for Map #8
>>
>> Wrote input for Map #9
>>
>> Wrote input for Map #10
>>
>> Wrote input for Map #11
>>
>> Wrote input for Map #12
>>
>> Wrote input for Map #13
>>
>> Wrote input for Map #14
>>
>> Wrote input for Map #15

Regards,

Yinghua

RE: Erro running pi programm

Kartashov, Andy 2012-11-09, 19:31

Try running "hostname -f" on each node, take a note of the fully qualified host address and replace your "master" with the your respective finding.

Here are my configuration files

core-site.xml

<configuration>
   <property>
   <name>fs.default.name<http://fs.default.name></name>
   <value>hdfs://master:9000</value>
   </property>
   <property>
   <name>hadoop.tmp.dir</name>
   <value>/usr/local/hadoop/tmp</value>
   </property>
</configuration>
mapred-site.xml

<configuration>
   <property>
   <name>mapreduce.framework.name<http://mapreduce.framework.name></name>
   <value>yarn</value>
   </property>

</configuration>

hdfs-site.xml

<configuration>
   <property>
   <name>dfs.replication</name>
   <value>2</value>
   </property>
   <property>
   <name>dfs.permissions</name>
   <value>false</value>
   </property>
   <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/yarn_data/hdfs/namenode</value>
   </property>
   <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/yarn_data/hdfs/datanode</value>
   </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>

  <value>mapreduce.shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
   <name>yarn.nodemanager.log-aggregation-enable</name>
   <value>true</value>
</property>
<property>
   <name>yarn.resourcemanager.resource-tracker.address</name>
   <value>master:8050</value>
</property>
<property>
   <name>yarn.resourcemanager.scheduler.address</name>
   <value>master:8030</value>
</property>
<property>
   <name>yarn.resourcemanager.scheduler.class</name>
   <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>
<property>
   <name>yarn.resourcemanager.address</name>
   <value>master:60400</value>
</property>

</configuration>

On Fri, Nov 9, 2012 at 9:51 AM, yinghua hu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi, Andy

Thanks for suggestions!

I am running it on a four node cluster on EC2. All the services started fine, Namenode, Datanode, ResourceManager, NodeManager and JobHistoryServer. Each node can ssh to all the nodes without problem.

But problem appears when trying to run any job.

From: Kartashov, Andy
Sent: Friday, November 09, 2012 12:37 PM
To: [EMAIL PROTECTED]
Subject: Erro running pi programm

Yinghua,

What mode are you running your hadoop in: Local/Pseud/Fully...?

Your hostname is not recognised

Your configuration setting seems to be wrong.

Hi, all

Could some help looking at this problem? I am setting up a four node cluster on EC2 and seems that the cluster is set up fine until I start testing.

I have tried password-less ssh from each node to all the nodes and there is no problem connecting. Any advice will be greatly appreciated!

[hduser@ip-XX-XX-XXX-XXX hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.4.jar pi -Dmapreduce.clientfactory.class.name<http://Dmapreduce.clientfactory.class.name>=org.apache.hadoop.mapred.YarnClientFactory -libjars share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.4.jar 16 10000
Number of Maps = 16
Samples per Map = 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Wrote input for Map #12
Wrote input for Map #13
Wrote input for Map #14
Wrote input for Map #15
Starting Job
12/11/09 12:02:59 INFO input.FileInputFormat: Total input paths to process : 16
12/11/09 12:02:59 INFO mapreduce.JobSubmitter: number of splits:16
12/11/09 12:02:59 WARN conf.Configuration: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
12/11/09 12:02:59 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
12/11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
12/11/09 12:02:59 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.job.name<http://mapred.job.name> is deprecated. Instead, use mapreduce.job.name<http://mapreduce.job.name>
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat

Re: Erro running pi programm

Ted Reynolds 2012-11-09, 20:43

yinghua,

One other thing to note here is that on EC2 computer are given two
hostnames, one internal and one external, you need to use the external.
These host names are listed in the ec2 configuration pane on the AWS
management console.
Ted Reynolds
Technical Support Engineer
Hortonworks
Work Phone: 408-645-7079

<http://hortonworks.com/download/>

On Fri, Nov 9, 2012 at 11:31 AM, Kartashov, Andy <[EMAIL PROTECTED]>wrote:

> Try running “hostname –f” on each node, take a note of the fully
> qualified host address and replace your “master” with the your respective
> finding.
>
>
>
> Here are my configuration files
>
>
>
> core-site.xml
>
>
>
> <configuration>
>
> <property>
>
> <name>fs.default.name</name>
>
> <value>hdfs://master:9000</value>
>
> </property>
>
> <property>
>
>
> <name>hadoop.tmp.dir</name>
>
>
> <value>/usr/local/hadoop/tmp</value>
>
>
> </property>
>
>
> </configuration>
>
>
>
>
>
> mapred-site.xml
>
>
>
> <configuration>
>
> <property>
>
> <name>mapreduce.framework.name</name>
>
> <value>yarn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
> hdfs-site.xml
>
>
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
> <property>
>
>
> <name>dfs.permissions</name>
>
>
> <value>false</value>
>
>
> </property>
>
>
> <property>
>
>
> <name>dfs.namenode.name.dir</name>
>
>
> <value>file:/home/hduser/yarn_data/hdfs/namenode</value>
>
>
> </property>
>
>
> <property>
>
>
> <name>dfs.datanode.data.dir</name>
>
>
> <value>file:/home/hduser/yarn_data/hdfs/datanode</value>
>
>
> </property>
>
>
> </configuration>
>
>
>
> yarn-site.xml
>
>
>
> <?xml version="1.0"?>
>
> <configuration>
>
> <property>
>
> <name>yarn.nodemanager.aux-services</name>
>
>
>
> <value>mapreduce.shuffle</value>
>
> </property>
>
> <property>
>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>
> </property>
>
> <property>
>
> <name>yarn.nodemanager.log-aggregation-enable</name>
>
> <value>true</value>
>
> </property>
>
> <property>
>
> <name>yarn.resourcemanager.resource-tracker.address</name>
>
> <value>master:8050</value>
>
> </property>
>
> <property>
>
> <name>yarn.resourcemanager.scheduler.address</name>
>
> <value>master:8030</value>
>
> </property>
>
> <property>
>
> <name>yarn.resourcemanager.scheduler.class</name>
>
>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>
>
>
> </property>
>
>
> <property>
>
>
> <name>yarn.resourcemanager.address</name>
>
>
> <value>master:60400</value>
>
>
> </property>
>
>
>
>
>
> </configuration>
>
>
>
> On Fri, Nov 9, 2012 at 9:51 AM, yinghua hu <[EMAIL PROTECTED]> wrote:
>
> Hi, Andy
>
>
>
> Thanks for suggestions!
>
>
>
> I am running it on a four node cluster on EC2. All the services started
> fine, Namenode, Datanode, ResourceManager, NodeManager and
> JobHistoryServer. Each node can ssh to all the nodes without problem.
>
>
>
> But problem appears when trying to run any job.
>
>
>
>
>
>
>
> *From:* Kartashov, Andy
> *Sent:* Friday, November 09, 2012 12:37 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Erro running pi programm
>
>
>
> Yinghua,
>
>
>
> What mode are you running your hadoop in: Local/Pseud/Fully...?
>
>
>
> Your hostname is not recognised
>
>
>
> Your configuration setting seems to be wrong.
>
>
>
>
>
>
>
>
>
>
>
> Hi, all
>
>
>
> Could some help looking at this problem? I am setting up a four node
> cluster on EC2 and seems that the cluster is set up fine until I start

Re: Erro running pi programm

yinghua hu 2012-11-10, 00:10

Hi, Andy and Ted

Thanks a lot for pointing out my silly error. The problem did go away after
I change the hostname from 'master' to the result of “hostname –f”.

Since I shutdown the EC2 instance every night, I thought that using
'master','slave1','slave2' will save typing after the full host name change
with reboot. But it did bring problems.

Thanks!

On Fri, Nov 9, 2012 at 12:43 PM, Ted Reynolds <[EMAIL PROTECTED]> wrote:

> yinghua,
>
> One other thing to note here is that on EC2 computer are given two
> hostnames, one internal and one external, you need to use the external.
> These host names are listed in the ec2 configuration pane on the AWS
> management console.
>
>
> Ted Reynolds
> Technical Support Engineer
> Hortonworks
> Work Phone: 408-645-7079
>
> <http://hortonworks.com/download/>
>
>
>
>
> On Fri, Nov 9, 2012 at 11:31 AM, Kartashov, Andy <[EMAIL PROTECTED]>wrote:
>
>> Try running “hostname –f” on each node, take a note of the fully
>> qualified host address and replace your “master” with the your respective
>> finding.
>>
>>
>>
>> Here are my configuration files
>>
>>
>>
>> core-site.xml
>>
>>
>>
>> <configuration>
>>
>> <property>
>>
>> <name>fs.default.name</name>
>>
>> <value>hdfs://master:9000</value>
>>
>> </property>
>>
>> <property>
>>
>>
>> <name>hadoop.tmp.dir</name>
>>
>>
>> <value>/usr/local/hadoop/tmp</value>
>>
>>
>> </property>
>>
>>
>> </configuration>
>>
>>
>>
>>
>>
>> mapred-site.xml
>>
>>
>>
>> <configuration>
>>
>> <property>
>>
>> <name>mapreduce.framework.name</name>
>>
>> <value>yarn</value>
>>
>> </property>
>>
>>
>>
>> </configuration>
>>
>>
>>
>> hdfs-site.xml
>>
>>
>>
>> <configuration>
>>
>> <property>
>>
>> <name>dfs.replication</name>
>>
>> <value>2</value>
>>
>> </property>
>>
>> <property>
>>
>>
>> <name>dfs.permissions</name>
>>
>>
>> <value>false</value>
>>
>>
>> </property>
>>
>>
>> <property>
>>
>>
>> <name>dfs.namenode.name.dir</name>
>>
>>
>> <value>file:/home/hduser/yarn_data/hdfs/namenode</value>
>>
>>
>> </property>
>>
>>
>> <property>
>>
>>
>> <name>dfs.datanode.data.dir</name>
>>
>>
>> <value>file:/home/hduser/yarn_data/hdfs/datanode</value>
>>
>>
>> </property>
>>
>>
>> </configuration>
>>
>>
>>
>> yarn-site.xml
>>
>>
>>
>> <?xml version="1.0"?>
>>
>> <configuration>
>>
>> <property>
>>
>> <name>yarn.nodemanager.aux-services</name>
>>
>>
>>
>> <value>mapreduce.shuffle</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>yarn.nodemanager.log-aggregation-enable</name>
>>
>> <value>true</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>yarn.resourcemanager.resource-tracker.address</name>
>>
>> <value>master:8050</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>yarn.resourcemanager.scheduler.address</name>
>>
>> <value>master:8030</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>yarn.resourcemanager.scheduler.class</name>
>>
>>
>>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>>
>>
>>
>> </property>
>>
>>
>> <property>
>>
>>
>> <name>yarn.resourcemanager.address</name>
>>
>>
>> <value>master:60400</value>
>>
>>
>> </property>
>>
>>
>>
>>
>>
>> </configuration>
>>
>>
>>
>> On Fri, Nov 9, 2012 at 9:51 AM, yinghua hu <[EMAIL PROTECTED]> wrote:
>>
>> Hi, Andy
>>
>>
>>
>> Thanks for suggestions!
>>
>>
>>
>> I am running it on a four node cluster on EC2. All the services started
>> fine, Namenode, Datanode, ResourceManager, NodeManager and
Regards,

Yinghua