Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: Strange machine behavior


Copy link to this message
-
Re: Strange machine behavior
On Sun, Dec 9, 2012 at 5:45 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I always set  "vm.swappiness = 0" for my hadoop servers (PostgreSQL
> servers too).
>

I have just done this for that machine.  So far, I have not seen a
re-occurrence of the strange behavior; it appears this might have solved
the problem.
> The reason is that Linux moves memory pages to swap space if they have not
> been accessed for a period of time (swapping).  Java virtual machine (JVM)
> does not act well in the case of swapping that will make MapReduce (and
> HBase and ZooKeeper) run into trouble.   So I would suggest to set
>  vm.swappiness = 0.
>
> Thanks
> ac
>
> On 9 Dec 2012, at 12:58 PM, seth wrote:
>
> > Oracle frequently recommends vm.swappiness = 0 to get well behaved RAC
> nodes.  Otherwise you start paging out things you don't usually want paged
> out in favor of a larger filesystem cache.
> >
> > There is also a vm parameter that controls the minimum size of the free
> chain, might want to increase that a bit.
> >
> > Also, look into hosting your JVM heap on huge pages, they can't be paged
> out and will help the JVM perform better too.
> >
> > On Dec 8, 2012, at 6:09 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:
> >
> >> Has anyone experienced a TaskTracker/DataNode behaving like the
> attached image?
> >>
> >> This was during a MR job (which runs often).  Note the extremely high
> System CPU time.  Upon investigating I saw that out of 64GB ram the system
> had allocated almost 45GB to cache!
> >>
> >> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync"
> which is roughly where the graph goes back to normal (much lower System,
> much higher User).
> >>
> >> This has happened a few times.
> >>
> >> I have tried playing with the sysctl vm.swappiness value (default of
> 60) by setting it to 30 (which it was at when the graph was collected) and
> now to 10.  I am not sure that helps.
> >>
> >> Any ideas?  Anyone else run into this before?
> >>
> >> 24 cores
> >> 64GB ram
> >> 4x2TB sata3 hdd
> >>
> >> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb
> heap) on this machine.
> >>
> >> 24 map slots (1gb heap each), no reducers.
> >>
> >> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.
> >> <cpu-use.png>
>
>
--

Robert Dyer
[EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB