Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Fault tolerance and Speculative Execution


Copy link to this message
-
Re: Fault tolerance and Speculative Execution
What you describe in the first paragraph is not true.

Speculative execution API toggles are listed in the documentation:
http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Job+Configuration
and in the mapred-default page in property form:
http://hadoop.apache.org/docs/stable/mapred-default.html. Speculative
execution is enabled by default.

On Thu, Jul 18, 2013 at 10:32 PM, Sundeep Kambhampati
<[EMAIL PROTECTED]> wrote:
> Hi all,
> Is it true that Hadoop 'always' starts same map tasks multiple times in
> order to be fault tolerant. i.e. same task is launched on several machines
> so that even if a node fails then same task would be available on other
> node. And in case no node fails redundant task that finishes late is killed.
> If it is true how can I change that configuration for Hadoop to do it or not
> do it.
>
> Speculative execution on the other hand does what I explained above
> (redundant map tasks) but only after all the map tasks are scheduled and if
> some nodes are free it starts redundant map tasks for those which are
> running slow. Is it always true? How do change this configuration
> enable/disable.
>
> I am using Hadoop-1.1.2 incase version matters.
>
> I really appreciate if someone could help me with this. Thank you.
>
> Regards
> Sundeep
>
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB