Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: What else can be built on top of YARN.


Copy link to this message
-
Re: What else can be built on top of YARN.
Thanks a lot for the responses. I now have a better understanding.

To answer to Jay's question , I think ZK can be used as for coordination
service for a distributed program (you have built it on top of exposed
granular api's) and it doesn't have features like resource management
(including allocation of resources based on requests) of cluster nodes ,
which yarn has.

Rahul
On Thu, May 30, 2013 at 5:59 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> What is the separation of concerns between YARN and Zookeeper?  That is,
> where does YARN leave off and where does Zookeeper begin?  Or is there some
> overlap....
>
>
> On Thu, May 30, 2013 at 2:42 AM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Rahul,
>>
>>   It is at least because of the reasons that Vinod listed that makes my
>> life easy for porting my application on to YARN instead of making it work
>> in the Map Reduce framework. The main purpose of me using YARN is to
>> exploit the resource management capabilities of YARN.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Wed, May 29, 2013 at 11:00 PM, Rahul Bhattacharjee <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Thanks for the response Krishna.
>>>
>>> I was wondering if it were possible for using MR to  solve you problem
>>> instead of building the whole stack on top of yarn.
>>> Most likely its not possible , thats why you are building it . I wanted
>>> to know why is that ?
>>>
>>> I am in just trying to find out the need or why we might need to write
>>> the application on yarn.
>>>
>>> Rahul
>>>
>>>
>>> On Wed, May 29, 2013 at 8:23 PM, Krishna Kishore Bonagiri <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Rahul,
>>>>
>>>>   I am porting a distributed application that runs on a fixed set of
>>>> given resources to YARN, with the aim of  being able to run it on a
>>>> dynamically selected resources whichever are available at the time of
>>>> running the application.
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Wed, May 29, 2013 at 8:04 PM, Rahul Bhattacharjee <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I was going through the motivation behind Yarn. Splitting the
>>>>> responsibility of JT is the major concern.Ultimately the base (Yarn) was
>>>>> built in a generic way for building other generic distributed applications
>>>>> too.
>>>>>
>>>>> I am not able to think of any other parallel processing use case that
>>>>> would be useful to built on top of YARN. I though of a lot of use cases
>>>>> that would be beneficial when run in parallel , but again ,we can do those
>>>>> using map only jobs in MR.
>>>>>
>>>>> Can someone tell me a scenario , where a application can utilize Yarn
>>>>> features or can be built on top of YARN and at the same time , it cannot be
>>>>> done efficiently using MRv2 jobs.
>>>>>
>>>>> thanks,
>>>>> Rahul
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB