Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Yarn -- one of the daemons getting killed


Copy link to this message
-
Re: Yarn -- one of the daemons getting killed
Hi Jeff,

  I have run the resource manager in the foreground without nohup and here
are the messages when it was killed, it says it is "Killed" but doesn't say
why!

13/12/17 03:14:54 INFO capacity.CapacityScheduler: Application
appattempt_1387266015651_0258_000001 released container
container_1387266015651_0258_01_000003 on node: host: isredeng:36576
#containers=2 available=7936 used=256 with event: FINISHED
13/12/17 03:14:54 INFO rmcontainer.RMContainerImpl:
container_1387266015651_0258_01_000005 Container Transitioned from ACQUIRED
to RUNNING
Killed
Thanks,
Kishore
On Mon, Dec 16, 2013 at 11:10 PM, Jeff Stuckman <[EMAIL PROTECTED]> wrote:

>  What if you open the daemons in a "screen" session rather than running
> them in the background -- for example, run "yarn resourcemanager". Then you
> can see exactly when they terminate, and hopefully why.
>
>    *From: *Krishna Kishore Bonagiri
> *Sent: *Monday, December 16, 2013 6:20 AM
> *To: *[EMAIL PROTECTED]
> *Reply To: *[EMAIL PROTECTED]
> *Subject: *Re: Yarn -- one of the daemons getting killed
>
>  Hi Vinod,
>
>   Yes, I am running on Linux.
>
>  I was actually searching for a corresponding message in /var/log/messages
> to confirm that OOM killed my daemons, but could not find any corresponding
> messages there! According to the following link, it looks like if it is a
> memory issue, I should see a messages even if OOM is disabled, but I don't
> see it.
>
>  http://www.redhat.com/archives/taroon-list/2007-August/msg00006.html
>
>    And, is memory consumption more in case of two node cluster than a
> single node one? Also, I see this problem only when I give "*" as the node
> name.
>
>    One other thing I suspected was the allowed number of user processes,
> I increased that to 31000 from 1024 but that also didn't help.
>
>  Thanks,
> Kishore
>
>
> On Fri, Dec 13, 2013 at 11:51 PM, Vinod Kumar Vavilapalli <
> [EMAIL PROTECTED]> wrote:
>
>> Yes, that is what I suspect. That is why I asked if everything is on a
>> single node. If you are running linux, linux OOM killer may be shooting
>> things down. When it happens, you will see something like "'killed process"
>> in system's syslog.
>>
>>    Thanks,
>> +Vinod
>>
>>  On Dec 13, 2013, at 4:52 AM, Krishna Kishore Bonagiri <
>> [EMAIL PROTECTED]> wrote:
>>
>>  Vinod,
>>
>>   One more thing I observed is that, my Client which submits Application
>> Master one after another continuously also gets killed sometimes. So, it is
>> always any of the Java Processes that is getting killed. Does it indicate
>> some excessive memory usage by them or something like that, that is causing
>> them die? If so, how can we resolve this kind of issue?
>>
>>  Thanks,
>> Kishore
>>
>>
>> On Fri, Dec 13, 2013 at 10:16 AM, Krishna Kishore Bonagiri <
>> [EMAIL PROTECTED]> wrote:
>>
>>> No, I am running on 2 node cluster.
>>>
>>>
>>> On Fri, Dec 13, 2013 at 1:52 AM, Vinod Kumar Vavilapalli <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Is all of this on a single node?
>>>>
>>>>   Thanks,
>>>> +Vinod
>>>>
>>>>  On Dec 12, 2013, at 3:26 AM, Krishna Kishore Bonagiri <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>  Hi,
>>>>   I am running a small application on YARN (2.2.0) in a loop of 500
>>>> times, and while doing so one of the daemons, node manager, resource
>>>> manager, or data node is getting killed (I mean disappearing) at a random
>>>> point. I see no information in the corresponding log files. How can I know
>>>> why is it happening so?
>>>>
>>>>   And, one more observation is that, this is happening only when I am
>>>> using "*" for node name in the container requests, otherwise when I used a
>>>> specific node name, everything is fine.
>>>>
>>>>  Thanks,
>>>> Kishore
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB