Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Dynamically set mapred.tasktracker.map.tasks.maximum from inside a job.


Copy link to this message
-
Re: Dynamically set mapred.tasktracker.map.tasks.maximum from inside a job.
The number of map tasks is determined by InputSplit.

On Wednesday, June 30, 2010, Pierre ANCELOT <[EMAIL PROTECTED]> wrote:
> Hi,
> Okay, so, if I set the 20 by default, I could maybe limit the number of
> concurrent maps per node instead?
> job.setNumReduceTasks exists but I see no equivalent for maps, though I
> think there was a setNumMapTasks before...
> Was it removed? Why?
> Any idea about how to acheive this?
>
> Thank you.
>
>
> On Wed, Jun 30, 2010 at 12:08 PM, Amareshwari Sri Ramadasu <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Pierre,
>>
>> "mapred.tasktracker.map.tasks.maximum" is a cluster level configuration,
>> cannot be set per job. It is loaded only while bringing up the TaskTracker.
>>
>> Thanks
>> Amareshwari
>>
>> On 6/30/10 3:05 PM, "Pierre ANCELOT" <[EMAIL PROTECTED]> wrote:
>>
>> Hi everyone :)
>> There's something I'm probably doing wrong but I can't seem to figure out
>> what.
>> I have two hadoop programs running one after the other.
>> This is done because they don't have the same needs in term of processor in
>> memory, so by separating them I optimize each task better.
>> Fact is, I need for the first job on every node
>> mapred.tasktracker.map.tasks.maximum set to 12.
>> For the second task, I need it to be set to 20.
>> so by default I set it to 12 and in the second job's code, I set this:
>>
>>        Configuration hadoopConfiguration = new Configuration();
>>        hadoopConfiguration.setInt("mapred.tasktracker.map.tasks.maximum",
>> 20);
>>
>> But when running the job, instead of having the 20 tasks on each node as
>> expected, I have 12....
>> Any idea please?
>>
>> Thank you.
>> Pierre.
>>
>>
>> --
>> http://www.neko-consulting.com
>> Ego sum quis ego servo
>> "Je suis ce que je protège"
>> "I am what I protect"
>>
>>
>
>
> --
> http://www.neko-consulting.com
> Ego sum quis ego servo
> "Je suis ce que je protège"
> "I am what I protect"
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB