Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - Hadoop MapReduce High Availability


Copy link to this message
-
Re: Hadoop MapReduce High Availability
Sandy Ryza 2013-04-29, 16:57
Hi Augusto,

In Hadoop 2, ResourceManager HA is being worked on under YARN-128 and
YARN-149.  There's a design doc for RM recovery on the latter.

Hadoop 1's MapReduce high availability story is kind of fragmented.
 Cloudera distribution has JobTracker HA based on the HA libraries
available in Hadoop 2.  I believe other distributions like Hortonworks' and
MapR's also have JobTracker HA solutions.  For a variety of reasons, none
of these are likely to make it into the Apache releases.

-Sandy

On Sun, Apr 28, 2013 at 2:52 PM, Augusto Souza <[EMAIL PROTECTED]>wrote:

> Hello,
>
> Sorry if this topic has already been discussed, but I am new to this
> mailing list and didn't find a way to check for past messages.
>
> Let me introduce myself. My name is Augusto Souza and I am a MSc
> student in Distributed Systems in University of Campinas (Brazil). One
> of the possibilities I have been thinking for developing my research
> is the problem of MapReduce High Availability.
>
> There are some open issues in Jira for this topic for quite a long time:
> https://issues.apache.org/jira/browse/MAPREDUCE-2288
> https://issues.apache.org/jira/browse/MAPREDUCE-225
>
> I also found some blog posts about this topic (eg:
>
> http://hortonworks.com/blog/high-availability-and-hadoop-1-0-perfect-together/
> ),
> but I didn't find one global and official solution from the community,
> is there one? Is there a way I could contribute with this? Are there
> some resources you guys recommend me to read about this topic?
>
> Thanks in advance.
>
> Best regards,
> Augusto Souza
>