Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: How to make AM terminate if client crashes?


+
Saeed Adel Mehraban 2014-01-12, 08:49
+
Vinod Kumar Vavilapalli 2014-01-13, 16:30
Copy link to this message
-
Re: How to make AM terminate if client crashes?
You would probably need to bake this into your own application. By default, a client never should need to keep an open active connection with the RM. It could keep an active connection with the AM ( application-specific code required ) but it would then also have to handle failover to a different AM if the first AM crashes and a new attempt is launched by the RM.

Does your application have access to a zookeeper cluster within your installation? It would be simple enough for someone to build a library which either uses an ephemeral node or updates some form of an applicationId specific keep-alive flag that the AM can monitor to make its decisions.

@Vinod, on a different note, does it make sense for the RM to run an embedded ZK to aid fencing across multiple AM attempts which could then be used by the above to piggyback on?

-- Hitesh

On Jan 13, 2014, at 8:30 AM, Vinod Kumar Vavilapalli wrote:

> The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
>
> Any more details on why this is desired?
>
> +Vinod
>
> On Jan 11, 2014, at 11:37 AM, John Lilley <[EMAIL PROTECTED]> wrote:
>
>> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes.  Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM?  Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
>> Thanks
>> John
+
John Lilley 2014-01-17, 14:19