Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Can container requests be made paralelly from multiple threads


Copy link to this message
-
Re: Can container requests be made paralelly from multiple threads
Hi Omkar,

 I have a distributed application that I am trying to port to YARN. My
application does many things in multiple threads in parallel, and those
threads in turn run some executables (how many of them depends on some
business logic and is variable). Now I am trying to launch those
executables on the containers acquired from the YARN's RM. If I could make
requests for containers from those parallel threads itself my life would be
much easier.
On Sat, Sep 28, 2013 at 1:36 AM, Omkar Joshi <[EMAIL PROTECTED]> wrote:

> My point is why you want multiple threads as a part of single AM talking
> to RM simultaneously? I think AMRMProtocol only AM is suppose to use and if
> the requirement is to have multiple requestor requesting resources then it
> should be clubbed into one single request and sent to RM. One more thing
> may be related to this. When AM makes a request to RM; today it gets
> resources only if earlier NM heartbeat resulted into RM scheduling one for
> it. So multiplexing AMRM requests wont help anyway but it will only
> complicate things on RM side. Scheduler is not kicked in (synchronously)
> when AM makes a request.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Fri, Sep 27, 2013 at 11:14 AM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Omkar,
>>
>>   Thanks for the quick reply. I have a requirement for sets of containers
>> depending on some of my business logic. I found that each of the request
>> allocations is taking around 2 seconds, so I am thinking of doing the
>> requests at the same from multiple threads.
>>
>> Kishore
>>
>>
>> On Fri, Sep 27, 2013 at 11:27 PM, Omkar Joshi <[EMAIL PROTECTED]>wrote:
>>
>>> Hi,
>>>
>>> I suggest you should not do that. After YARN-744 goes in this will be
>>> prevented on RM side. May I know why you want to do this? any advantage/
>>> use case?
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Fri, Sep 27, 2013 at 8:31 AM, Krishna Kishore Bonagiri <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>>   Can we submit container requests from multiple threads in parallel to
>>>> the Resource Manager?
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB