Hi,
 
I wanted to know why I see running Child processes on my datanodes even though
there is no job running at that time. Are these left over from failed attempts?

Is there anything I can do to keep these clean?

Thanks,
Deepika
      

Use jps to find out pid of the Child.
Then use this to find out which job the Child belongs to:
ps aux | grep <pid>

On Wed, Aug 25, 2010 at 12:20 PM, C J <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even
> though
> there is no job running at that time. Are these left over from failed
> attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>

Thanks for your reply.

Some of these child tasks belong to successful jobs. I am wondering why they are
still hanging there for long finished jobs.

________________________________
From: Ted Yu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wed, August 25, 2010 4:17:38 PM
Subject: Re: Child processes on datanodes/task trackers

Use jps to find out pid of the Child.
Then use this to find out which job the Child belongs to:
ps aux | grep <pid>

On Wed, Aug 25, 2010 at 12:20 PM, C J <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even
> though
> there is no job running at that time. Are these left over from failed
> attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>

      

After you obtain pid, you can use jstack to see what the Child process was
doing.

What hadoop version are you using ?

On Wed, Aug 25, 2010 at 7:28 PM, C J <[EMAIL PROTECTED]> wrote:

> Thanks for your reply.
>
> Some of these child tasks belong to successful jobs. I am wondering why
> they are
> still hanging there for long finished jobs.
>
>
>
>
>
> ________________________________
> From: Ted Yu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wed, August 25, 2010 4:17:38 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> Use jps to find out pid of the Child.
> Then use this to find out which job the Child belongs to:
> ps aux | grep <pid>
>
> On Wed, Aug 25, 2010 at 12:20 PM, C J <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I wanted to know why I see running Child processes on my datanodes even
> > though
> > there is no job running at that time. Are these left over from failed
> > attempts?
> >
> > Is there anything I can do to keep these clean?
> >
> > Thanks,
> > Deepika
> >
> >
> >
>
>
>
>
>

Thanks Ted!

I did a jstack and it seems there is an issue with ehcache that I am using in
the mapper task.
"net.sf.ehcache.CacheManager@57ac3379" daemon prio=10 tid=0x0000000059180800
nid=0x379e in Object.wait() [0x0000000041506000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None
.
.
.
The hadoop version I am using is 0.20.2.

Thanks.

________________________________
From: Ted Yu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wed, August 25, 2010 7:34:35 PM
Subject: Re: Child processes on datanodes/task trackers

After you obtain pid, you can use jstack to see what the Child process was
doing.

What hadoop version are you using ?

On Wed, Aug 25, 2010 at 7:28 PM, C J <[EMAIL PROTECTED]> wrote:

> Thanks for your reply.
>
> Some of these child tasks belong to successful jobs. I am wondering why
> they are
> still hanging there for long finished jobs.
>
>
>
>
>
> ________________________________
> From: Ted Yu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wed, August 25, 2010 4:17:38 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> Use jps to find out pid of the Child.
> Then use this to find out which job the Child belongs to:
> ps aux | grep <pid>
>
> On Wed, Aug 25, 2010 at 12:20 PM, C J <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I wanted to know why I see running Child processes on my datanodes even
> > though
> > there is no job running at that time. Are these left over from failed
> > attempts?
> >
> > Is there anything I can do to keep these clean?
> >
> > Thanks,
> > Deepika
> >
> >
> >
>
>
>
>
>

      

I don't use ehcache.
Did you forget to close CacheManager at the end of your job by any chance ?

On Wed, Aug 25, 2010 at 7:59 PM, C J <[EMAIL PROTECTED]> wrote:

> Thanks Ted!
>
> I did a jstack and it seems there is an issue with ehcache that I am using
> in
> the mapper task.
>
>
> "net.sf.ehcache.CacheManager@57ac3379" daemon prio=10
> tid=0x0000000059180800
> nid=0x379e in Object.wait() [0x0000000041506000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
>        at java.util.TimerThread.mainLoop(Timer.java:509)
>        - locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
>        at java.util.TimerThread.run(Timer.java:462)
>
>   Locked ownable synchronizers:
>        - None
> .
> .
> .
>
>
> The hadoop version I am using is 0.20.2.
>
> Thanks.
>
>
>
> ________________________________
> From: Ted Yu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wed, August 25, 2010 7:34:35 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> After you obtain pid, you can use jstack to see what the Child process was
> doing.
>
> What hadoop version are you using ?
>
> On Wed, Aug 25, 2010 at 7:28 PM, C J <[EMAIL PROTECTED]> wrote:
>
> > Thanks for your reply.
> >
> > Some of these child tasks belong to successful jobs. I am wondering why
> > they are
> > still hanging there for long finished jobs.
> >
> >
> >
> >
> >
> > ________________________________
> > From: Ted Yu <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Sent: Wed, August 25, 2010 4:17:38 PM
> > Subject: Re: Child processes on datanodes/task trackers
> >
> > Use jps to find out pid of the Child.
> > Then use this to find out which job the Child belongs to:
> > ps aux | grep <pid>
> >
> > On Wed, Aug 25, 2010 at 12:20 PM, C J <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > I wanted to know why I see running Child processes on my datanodes even
> > > though
> > > there is no job running at that time. Are these left over from failed
> > > attempts?
> > >
> > > Is there anything I can do to keep these clean?
> > >
> > > Thanks,
> > > Deepika
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>

On 25/08/10 20:20, C J wrote:
> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even though
> there is no job running at that time. Are these left over from failed attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>

could be a bug somewhere in Hadoop or your code.

1. jps -v gives you the process ids,

2. jstack -l <pid> gives you the stack dumps and locks for the given
process, so you can see where the threads are -this will help you track
down the problem

3. It is, sadly, going to be you who gets to track down the problem. But
if you conclude that it is hadoop and not your code at fault, those
stack traces should be attached to the JIRA issue you can create

Thanks Ted. Yes, I forgot to shutdown the CacheManager.

Appreciate your help.
________________________________
From: Ted Yu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wed, August 25, 2010 8:39:01 PM
Subject: Re: Child processes on datanodes/task trackers

I don't use ehcache.
Did you forget to close CacheManager at the end of your job by any chance ?

On Wed, Aug 25, 2010 at 7:59 PM, C J <[EMAIL PROTECTED]> wrote:

> Thanks Ted!
>
> I did a jstack and it seems there is an issue with ehcache that I am using
> in
> the mapper task.
>
>
> "net.sf.ehcache.CacheManager@57ac3379" daemon prio=10
> tid=0x0000000059180800
> nid=0x379e in Object.wait() [0x0000000041506000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
>        at java.util.TimerThread.mainLoop(Timer.java:509)
>        - locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
>        at java.util.TimerThread.run(Timer.java:462)
>
>   Locked ownable synchronizers:
>        - None
> .
> .
> .
>
>
> The hadoop version I am using is 0.20.2.
>
> Thanks.
>
>
>
> ________________________________
> From: Ted Yu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wed, August 25, 2010 7:34:35 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> After you obtain pid, you can use jstack to see what the Child process was
> doing.
>
> What hadoop version are you using ?
>
> On Wed, Aug 25, 2010 at 7:28 PM, C J <[EMAIL PROTECTED]> wrote:
>
> > Thanks for your reply.
> >
> > Some of these child tasks belong to successful jobs. I am wondering why
> > they are
> > still hanging there for long finished jobs.
> >
> >
> >
> >
> >
> > ________________________________
> > From: Ted Yu <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Sent: Wed, August 25, 2010 4:17:38 PM
> > Subject: Re: Child processes on datanodes/task trackers
> >
> > Use jps to find out pid of the Child.
> > Then use this to find out which job the Child belongs to:
> > ps aux | grep <pid>
> >
> > On Wed, Aug 25, 2010 at 12:20 PM, C J <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > I wanted to know why I see running Child processes on my datanodes even
> > > though
> > > there is no job running at that time. Are these left over from failed
> > > attempts?
> > >
> > > Is there anything I can do to keep these clean?
> > >
> > > Thanks,
> > > Deepika
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>