I have a check monitoring the page jobtracker:50030/jobtracker.jsp,
and the check shows timeout (180 sec) pretty often.
Once I jump and browse to the page it actually take me from 5 sec to 5
minutes to render.

the heap size is 12G. from top command it is not used up.

CPU load is 1 ish (in all 5, 10, 15 load)

disk is far from full

Swap is untouched
Any input for how to troubleshoot this will be useful and greatly grateful

Thanks
Patai

Does this happen all of the time?

I've seen cases where if I have a job or process that is hosing the CPU of
a tasktracker, it can cause the job tracker to pause while trying to
contact that tasktracker.  Once the CPU load dips down to acceptable levels
communication can flow again and the job tracker resumes.  That could be
happening to you?  When it happens, requests to the web ui for that
tasktracker also don't work.

This usually happened for me when I was running too many concurrent tasks
on each tasktracker, though it actually came down to saturating the disk
and having a bunch of iowait causing processes to pile up.  Lowering the
tasks per tasktracker caused less contention for disk which freed up the
iowait and the waiting processes.  If your tasks are not disk heavy you can
put more tasks on the tasktracker, but we had a few disk-heavy jobs and
only when those ran did we see this.
On Thu, Nov 1, 2012 at 4:47 PM, Patai Sangbutsarakum <
[EMAIL PROTECTED]> wrote:

> I have a check monitoring the page jobtracker:50030/jobtracker.jsp,
> and the check shows timeout (180 sec) pretty often.
> Once I jump and browse to the page it actually take me from 5 sec to 5
> minutes to render.
>
> the heap size is 12G. from top command it is not used up.
>
> CPU load is 1 ish (in all 5, 10, 15 load)
>
> disk is far from full
>
> Swap is untouched
>
>
> Any input for how to troubleshoot this will be useful and greatly grateful
>
> Thanks
> Patai
>

Hi Patai,
Have you looked into verifying if it is network related, maybe see what the
ping responses are to that node?
On Thu, Nov 1, 2012 at 1:47 PM, Patai Sangbutsarakum <
[EMAIL PROTECTED]> wrote:

> I have a check monitoring the page jobtracker:50030/jobtracker.jsp,
> and the check shows timeout (180 sec) pretty often.
> Once I jump and browse to the page it actually take me from 5 sec to 5
> minutes to render.
>
> the heap size is 12G. from top command it is not used up.
>
> CPU load is 1 ish (in all 5, 10, 15 load)
>
> disk is far from full
>
> Swap is untouched
>
>
> Any input for how to troubleshoot this will be useful and greatly grateful
>
> Thanks
> Patai
>

Do you run a TaskTracker on your JobTracker machine?

On Fri, Nov 2, 2012 at 2:17 AM, Patai Sangbutsarakum
<[EMAIL PROTECTED]> wrote:
> I have a check monitoring the page jobtracker:50030/jobtracker.jsp,
> and the check shows timeout (180 sec) pretty often.
> Once I jump and browse to the page it actually take me from 5 sec to 5
> minutes to render.
>
> the heap size is 12G. from top command it is not used up.
>
> CPU load is 1 ish (in all 5, 10, 15 load)
>
> disk is far from full
>
> Swap is untouched
>
>
> Any input for how to troubleshoot this will be useful and greatly grateful
>
> Thanks
> Patai

--
Harsh J

What about the gc logs?

On Fri, Nov 2, 2012 at 11:16 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Do you run a TaskTracker on your JobTracker machine?
>
> On Fri, Nov 2, 2012 at 2:17 AM, Patai Sangbutsarakum
> <[EMAIL PROTECTED]> wrote:
> > I have a check monitoring the page jobtracker:50030/jobtracker.jsp,
> > and the check shows timeout (180 sec) pretty often.
> > Once I jump and browse to the page it actually take me from 5 sec to 5
> > minutes to render.
> >
> > the heap size is 12G. from top command it is not used up.
> >
> > CPU load is 1 ish (in all 5, 10, 15 load)
> >
> > disk is far from full
> >
> > Swap is untouched
> >
> >
> > Any input for how to troubleshoot this will be useful and greatly
> grateful
> >
> > Thanks
> > Patai
>
>
>
> --
> Harsh J
>

... another thought... do you happened to have some ungracefully  terminated jobs still running in the background..
Try "hadoop job -list"

Sometimes, when I hardstop a job  and restart a new one, I notice a slow down until I kill those jobs gracefully by running "hadoop job -kill <job-id> ". The job-ids are displayed by that previous command.
From: Robert Molina [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 01, 2012 5:31 PM
To: [EMAIL PROTECTED]
Subject: Re: jobtracker page @50030 timeout or take very long time.

Hi Patai,
Have you looked into verifying if it is network related, maybe see what the ping responses are to that node?
On Thu, Nov 1, 2012 at 1:47 PM, Patai Sangbutsarakum <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
I have a check monitoring the page jobtracker:50030/jobtracker.jsp,
and the check shows timeout (180 sec) pretty often.
Once I jump and browse to the page it actually take me from 5 sec to 5
minutes to render.

the heap size is 12G. from top command it is not used up.

CPU load is 1 ish (in all 5, 10, 15 load)

disk is far from full

Swap is untouched
Any input for how to troubleshoot this will be useful and greatly grateful

Thanks
Patai

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel