Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Distributing the code to multiple nodes


Copy link to this message
-
Re: Distributing the code to multiple nodes
Ashish Jain 2014-01-15, 08:37
Hello Sudhakara,

Thanks for your suggestion. However once I change the mapreduce framework
to yarn my map reduce jobs does not get executed at all. It seems it is
waiting on some thread indefinitely. Here is what I have done

1) Set the mapreduce framework to yarn in mapred-site.xml
<property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
</property>
2) Run the example again using the command
./hadoop dfs wordCount.jar /opt/ApacheHadoop/temp/worker.log
/opt/ApacheHadoop/out/

The jobs are just stuck and do not move further.
I also tried the following and it complains of filenotfound exception and
some security exception

./hadoop dfs wordCount.jar file:///opt/ApacheHadoop/temp/worker.log
file:///opt/ApacheHadoop/out/

Below is the status of the job from hadoop application console. The
progress bar does not move at all.

ID
User
Name
Application Type
Queue
StartTime
FinishTime
State
FinalStatus
Progress
Tracking UI
application_1389771586883_0002<http://10.12.11.210:8088/cluster/app/application_1389771586883_0002>
rootwordcountMAPREDUCEdefaultWed, 15 Jan 2014 07:52:04 GMTN/AACCEPTED
UNDEFINED

UNASSIGNE <http://10.12.11.210:8088/cluster/apps#>
Please advice what should I do

--Ashish
On Tue, Jan 14, 2014 at 5:48 PM, sudhakara st <[EMAIL PROTECTED]>wrote:

> Hello Ashish
> It seems job is running in Local job runner(LocalJobRunner) by reading the
> Local file system. Can you try by give the full URI path of the input and
> output path.
> like
> $hadoop jar program.jar   ProgramName -Dmapreduce.framework.name=yarn
> file:///home/input/  file:///home/output/
>
>
> On Mon, Jan 13, 2014 at 3:02 PM, Ashish Jain <[EMAIL PROTECTED]> wrote:
>
>> German,
>>
>> This does not seem to be helping. I tried to use the Fairscheduler as my
>> resource manger but the behavior remains same. I could see the
>> fairscheduler log getting continuous heart beat from both the other nodes.
>> But it is still not distributing the work to other nodes. What I did next
>> was started 3 jobs simultaneously so that may be some part of one of the
>> job be distributed to other nodes. However still only one node is being
>> used :(((. What is that is going wrong can some one help?
>>
>> Sample of fairsheduler log:
>> 2014-01-13 15:13:54,293 HEARTBEAT       l1dev-211
>> 2014-01-13 15:13:54,953 HEARTBEAT       l1-dev06
>> 2014-01-13 15:13:54,988 HEARTBEAT       l1-DEV05
>> 2014-01-13 15:13:55,295 HEARTBEAT       l1dev-211
>> 2014-01-13 15:13:55,956 HEARTBEAT       l1-dev06
>> 2014-01-13 15:13:55,993 HEARTBEAT       l1-DEV05
>> 2014-01-13 15:13:56,297 HEARTBEAT       l1dev-211
>> 2014-01-13 15:13:56,960 HEARTBEAT       l1-dev06
>> 2014-01-13 15:13:56,997 HEARTBEAT       l1-DEV05
>> 2014-01-13 15:13:57,299 HEARTBEAT       l1dev-211
>> 2014-01-13 15:13:57,964 HEARTBEAT       l1-dev06
>> 2014-01-13 15:13:58,001 HEARTBEAT       l1-DEV05
>>
>> My Data distributed as blocks to other nodes. The host with IP
>> 10.12.11.210 has all the data and this is the one which is serving all the
>> request.
>>
>> Total number of blocks: 8
>> 1073741866:         10.12.11.211:50010    View Block Info
>> 10.12.11.210:50010    View Block Info
>> 1073741867:         10.12.11.211:50010    View Block Info
>> 10.12.11.210:50010    View Block Info
>> 1073741868:         10.12.11.210:50010    View Block Info
>> 10.12.11.209:50010    View Block Info
>> 1073741869:         10.12.11.210:50010    View Block Info
>> 10.12.11.209:50010    View Block Info
>> 1073741870:         10.12.11.211:50010    View Block Info
>> 10.12.11.210:50010    View Block Info
>> 1073741871:         10.12.11.210:50010    View Block Info
>> 10.12.11.209:50010    View Block Info
>> 1073741872:         10.12.11.211:50010    View Block Info
>> 10.12.11.210:50010    View Block Info
>> 1073741873:         10.12.11.210:50010    View Block Info
>> 10.12.11.209:50010    View Block Info
>>
>> Someone please advice on how to go about this.
>>
>> --Ashish
>>
>>
>> On Fri, Jan 10, 2014 at 12:58 PM, Ashish Jain <[EMAIL PROTECTED]> wrote: