Mark Kerzner
2012-11-13, 15:16
Mark Kerzner
2012-11-13, 15:19
Mark Kerzner
2012-11-13, 15:25
Kartashov, Andy
2012-11-13, 15:26
Serge Blazhiyevskyy
2012-11-13, 15:27
Jay Vyas
2012-11-13, 15:27
Mark Kerzner
2012-11-13, 15:50
Mark Kerzner
2012-11-13, 15:52
-
mapred.tasktracker.map.tasks.maximumMark Kerzner 2012-11-13, 15:16
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value
for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this
situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
I have a cluster with 4 nodes and 32 many cores on each. My default value
for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this
situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
-
Re: mapred.tasktracker.map.tasks.maximumMark Kerzner 2012-11-13, 15:19
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
[EMAIL PROTECTED]> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>
> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
[EMAIL PROTECTED]> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>
> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
-
Re: mapred.tasktracker.map.tasks.maximumMark Kerzner 2012-11-13, 15:25
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
[EMAIL PROTECTED]> wrote:
> Look on the job tracker web UI is parameter has taken effect (port 50030)
>
> It might be in the wrong config file or something
>
>
> Serge
>
> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>
> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Date: Tuesday, November 13, 2012 5:19 PM
> To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Subject: Re: mapred.tasktracker.map.tasks.maximum
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>>
> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]
> ><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
[EMAIL PROTECTED]> wrote:
> Look on the job tracker web UI is parameter has taken effect (port 50030)
>
> It might be in the wrong config file or something
>
>
> Serge
>
> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>
> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Date: Tuesday, November 13, 2012 5:19 PM
> To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Subject: Re: mapred.tasktracker.map.tasks.maximum
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>>
> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]
> ><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
-
RE: mapred.tasktracker.map.tasks.maximumKartashov, Andy 2012-11-13, 15:26
Mark,
The way I understand it...
# of mappers is calculated by deviding your input by file split size (64Mb default).
So, say your input is 64Gb in size so you will end up with 1000 mappers.
Slots are a different story. It is number of map tasks processed by a single node in parrallel. It is normally calculated by the number of cores that node is running on (one per core).
Rgds,
From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, November 13, 2012 10:20 AM
To: [EMAIL PROTECTED]
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
The way I understand it...
# of mappers is calculated by deviding your input by file split size (64Mb default).
So, say your input is 64Gb in size so you will end up with 1000 mappers.
Slots are a different story. It is number of map tasks processed by a single node in parrallel. It is normally calculated by the number of cores that node is running on (one per core).
Rgds,
From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, November 13, 2012 10:20 AM
To: [EMAIL PROTECTED]
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
-
Re: mapred.tasktracker.map.tasks.maximumSerge Blazhiyevskyy 2012-11-13, 15:27
;)
From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Tuesday, November 13, 2012 5:25 PM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Tuesday, November 13, 2012 5:25 PM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
-
Re: mapred.tasktracker.map.tasks.maximumJay Vyas 2012-11-13, 15:27
Hmmm What do you mean wrong configuration file.? How could that ever happen?
Jay Vyas
On Nov 13, 2012, at 10:25 AM, Mark Kerzner <[EMAIL PROTECTED]> wrote:
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]> wrote:
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>
Jay Vyas
On Nov 13, 2012, at 10:25 AM, Mark Kerzner <[EMAIL PROTECTED]> wrote:
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]> wrote:
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>
-
Re: mapred.tasktracker.map.tasks.maximumMark Kerzner 2012-11-13, 15:50
I am using a Hadoop manager, and it stores the values in a database,
outside the configuration files.
On Tue, Nov 13, 2012 at 10:27 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> Hmmm What do you mean wrong configuration file.? How could that ever
> happen?
>
> Jay Vyas
>
> On Nov 13, 2012, at 10:25 AM, Mark Kerzner <[EMAIL PROTECTED]>
> wrote:
>
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
> [EMAIL PROTECTED]> wrote:
>
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
>> [EMAIL PROTECTED]>>
>> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
>> [EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:
>> [EMAIL PROTECTED]>>>
>> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]
>> ><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value
>> for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
>> it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this
>> situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>>
>>
>
outside the configuration files.
On Tue, Nov 13, 2012 at 10:27 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> Hmmm What do you mean wrong configuration file.? How could that ever
> happen?
>
> Jay Vyas
>
> On Nov 13, 2012, at 10:25 AM, Mark Kerzner <[EMAIL PROTECTED]>
> wrote:
>
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
> [EMAIL PROTECTED]> wrote:
>
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
>> [EMAIL PROTECTED]>>
>> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
>> [EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:
>> [EMAIL PROTECTED]>>>
>> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:
>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]
>> ><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value
>> for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
>> it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this
>> situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>>
>>
>
-
Re: mapred.tasktracker.map.tasks.maximumMark Kerzner 2012-11-13, 15:52
An important distinction, max slots per node is one thing, and total
mappers is something else - it is the total that ran. But now I know where
to look.
thank you,
mark
On Tue, Nov 13, 2012 at 10:26 AM, Kartashov, Andy <[EMAIL PROTECTED]>wrote:
> Mark,
>
>
>
> The way I understand it…
>
>
>
> # of mappers is calculated by deviding your input by file split size (64Mb
> default).
>
>
>
> So, say your input is 64Gb in size so you will end up with 1000 mappers.
>
>
>
> Slots are a different story. It is number of map tasks processed by a
> single node in parrallel. It is normally calculated by the number of cores
> that node is running on (one per core).
>
>
>
> Rgds,
>
>
>
> *From:* Mark Kerzner [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, November 13, 2012 10:20 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: mapred.tasktracker.map.tasks.maximum
>
>
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> [EMAIL PROTECTED]> wrote:
>
> What hadoop version are we talking about?
>
> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>
> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
mappers is something else - it is the total that ran. But now I know where
to look.
thank you,
mark
On Tue, Nov 13, 2012 at 10:26 AM, Kartashov, Andy <[EMAIL PROTECTED]>wrote:
> Mark,
>
>
>
> The way I understand it…
>
>
>
> # of mappers is calculated by deviding your input by file split size (64Mb
> default).
>
>
>
> So, say your input is 64Gb in size so you will end up with 1000 mappers.
>
>
>
> Slots are a different story. It is number of map tasks processed by a
> single node in parrallel. It is normally calculated by the number of cores
> that node is running on (one per core).
>
>
>
> Rgds,
>
>
>
> *From:* Mark Kerzner [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, November 13, 2012 10:20 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: mapred.tasktracker.map.tasks.maximum
>
>
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> [EMAIL PROTECTED]> wrote:
>
> What hadoop version are we talking about?
>
> From: Mark Kerzner <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>
> Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
