-Re: Strange machine behavior
Robert Dyer 2012-12-11, 03:33
On Sun, Dec 9, 2012 at 5:45 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> I always set "vm.swappiness = 0" for my hadoop servers (PostgreSQL
> servers too).
I have just done this for that machine. So far, I have not seen a
re-occurrence of the strange behavior; it appears this might have solved
> The reason is that Linux moves memory pages to swap space if they have not
> been accessed for a period of time (swapping). Java virtual machine (JVM)
> does not act well in the case of swapping that will make MapReduce (and
> HBase and ZooKeeper) run into trouble. So I would suggest to set
> vm.swappiness = 0.
> On 9 Dec 2012, at 12:58 PM, seth wrote:
> > Oracle frequently recommends vm.swappiness = 0 to get well behaved RAC
> nodes. Otherwise you start paging out things you don't usually want paged
> out in favor of a larger filesystem cache.
> > There is also a vm parameter that controls the minimum size of the free
> chain, might want to increase that a bit.
> > Also, look into hosting your JVM heap on huge pages, they can't be paged
> out and will help the JVM perform better too.
> > On Dec 8, 2012, at 6:09 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:
> >> Has anyone experienced a TaskTracker/DataNode behaving like the
> attached image?
> >> This was during a MR job (which runs often). Note the extremely high
> System CPU time. Upon investigating I saw that out of 64GB ram the system
> had allocated almost 45GB to cache!
> >> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync"
> which is roughly where the graph goes back to normal (much lower System,
> much higher User).
> >> This has happened a few times.
> >> I have tried playing with the sysctl vm.swappiness value (default of
> 60) by setting it to 30 (which it was at when the graph was collected) and
> now to 10. I am not sure that helps.
> >> Any ideas? Anyone else run into this before?
> >> 24 cores
> >> 64GB ram
> >> 4x2TB sata3 hdd
> >> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb
> heap) on this machine.
> >> 24 map slots (1gb heap each), no reducers.
> >> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.
> >> <cpu-use.png>