Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> What resources are used by idle NameNode and JobTracker tasks


Copy link to this message
-
Re: What resources are used by idle NameNode and JobTracker tasks
The NN has a constant memory use and the CPU usage should also be
patterned in idle mode. The JT usually retires jobs over the days, and
the memory and CPU usage should both go down over time in idle mode,
especially if you indicate a week of idleness. There is however the DN
block scanner that may trigger itself and begin consuming a bit of an
IO in verifying block data integrity (to re-populate rot/etc.
corrupted blocks on disks), which would probably be the only usage
registered on an idle cluster with no users talking to it.

What exact 'resources' was the complaint about? Having more idea on
that would help us trace or answer better.

On Mon, Feb 11, 2013 at 11:37 PM, Steve Lewis <[EMAIL PROTECTED]> wrote:
> I have Hadoop running on 8 nodes in a 50 node cluster. The NameNode and
> Jobtrackers have been running for months while jobs were run occasionally -
> usually no more than a dozen a week and more typically none. Recently the
> cluster manager accused Hadoop of consuming a large number of resources in
> the cluster - this occurred during a week an which I was running no jobs.
> While I stopped Hadoop  the trackers have been running for many months
> stably.
>   Is there any reason why either of these jobs should suddenly and without
> jobs being run increase their consumption of resources in a serious way??
>
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>

--
Harsh J