Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - mapred.tasktacker.reduce.tasks.maximum issue


+
Vincent Y. Shen 2013-11-14, 14:53
+
Dieter De Witte 2013-11-14, 14:56
+
Vincent Y. Shen 2013-11-14, 14:58
+
Dieter De Witte 2013-11-14, 15:02
+
Vincent Y. Shen 2013-11-14, 15:53
Copy link to this message
-
Re: mapred.tasktacker.reduce.tasks.maximum issue
Zizon Qiu 2013-11-15, 01:57
may be it is not reading the right configuration file.

if possible,make a malformed file,then restart tasktracker where it must
fail to do so.
On Thu, Nov 14, 2013 at 11:53 PM, Vincent Y. Shen
<[EMAIL PROTECTED]>wrote:

> Hmmmm i tried but still not working... 4 nodes 8 reducers.... all values
> are final
>
>
> On Thu, Nov 14, 2013 at 3:02 PM, Dieter De Witte <[EMAIL PROTECTED]>wrote:
>
>> you could make the values final: add <final> true </final> to make sure
>> they cannot be overridden..
>>
>>
>> 2013/11/14 Vincent Y. Shen <[EMAIL PROTECTED]>
>>
>>> Hi Dieter,
>>>
>>> Thanks a lot for prompt reply! very much appreciated.
>>>
>>> Indeed, I did change in the mapred-site.xml in ALL the nodes and
>>> restarted the cluster 4 times but no luck...
>>>
>>>
>>> Any other hint?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 14, 2013 at 2:56 PM, Dieter De Witte <[EMAIL PROTECTED]>wrote:
>>>
>>>> Dear Vincent,
>>>>
>>>> You have to make sure that mapred-site is modified on every tasktracker
>>>> node! Not just the master! Mapred-site can be customized if you have a
>>>> heterogeneous cluster. I usually start the dfs, then copy all the
>>>> mapredsites to the slave nodes, then run start-mapred.sh and then start a
>>>> job.
>>>>
>>>> Regards, Dieter
>>>>
>>>>
>>>> 2013/11/14 Vincent Y. Shen <[EMAIL PROTECTED]>
>>>>
>>>> Hi Guys,
>>>>>
>>>>> I am increasing mapred.tasktacker.reduce.tasks.maximum in the hadoop
>>>>> cluster. But regardless whatever I tried, hadoop refuses to use my value
>>>>> but the default (2)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> What I did do
>>>>> ==========>>>>>
>>>>> stop mapred
>>>>> stop dfs
>>>>>
>>>>> update the value of mapred.tasktacker.reduce.tasks.maximum
>>>>>
>>>>>
>>>>> start dfs
>>>>> start mapred
>>>>>
>>>>>
>>>>> But the system still uses default values
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Environment
>>>>> ========>>>>>
>>>>> Hadoop version: 1.1.2
>>>>> Java version: 7
>>>>>
>>>>>
>>>>> Mapred-site.xml
>>>>> ===========>>>>>
>>>>> <property>
>>>>>   <name>mapred.system.dir</name>
>>>>>   <value>/hadoop/mapred/system</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>>   <name>mapred.reduce.tasks</name>
>>>>>   <value>50</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>>   <name>mapred.tasktacker.reduce.tasks.maximum</name>
>>>>>   <value>15</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>>   <name>mapred.tasktacker.map.tasks.maximum</name>
>>>>>   <value>15</value>
>>>>> </property>
>>>>>
>>>>>
>>>>> <property>
>>>>>   <name>mapred.child.java.opts</name>
>>>>>   <value>-Xmx500m</value>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> <property>
>>>>>   <name>mapred.job.tracker</name>
>>>>>   <value>machine1.mydomain.com:54311</value>
>>>>>
>>>>> </property>
>>>>>
>>>>>
>>>>>
>>>>> Thanks guys for any pointers!!!!
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Vincent.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
+
Adam Kawa 2013-11-27, 23:43