Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: Strange machine behavior


+
Robert Dyer 2012-12-11, 03:33
+
Andy Isaacson 2012-12-10, 19:23
+
Robert Dyer 2012-12-11, 03:30
Copy link to this message
-
Re: Strange machine behavior
Are you seeing any performance impact with this cache increase? It is normal in linux system to grab high cache level.
-Bharath
________________________________
 From: Andy Isaacson <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, December 10, 2012 11:23 AM
Subject: Re: Strange machine behavior
 
What kernel did you see this on? Was there significant swap traffic
(si/so in vmstat output) during the high-system-time period?

BTW, you don't need to nor do you want to run sync(1) when
manipulating drop_caches, it just causes additional noise and
slowdown. drop_caches doesn't have any impact on correctness; it won't
cause data loss (by dropping a dirty page or whatever). I've had sync
calls take 10 minutes to complete, so the unnecessary impact can be
significant.

-andy

On Sat, Dec 8, 2012 at 4:09 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:
> Has anyone experienced a TaskTracker/DataNode behaving like the attached
> image?
>
> This was during a MR job (which runs often).  Note the extremely high System
> CPU time.  Upon investigating I saw that out of 64GB ram the system had
> allocated almost 45GB to cache!
>
> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync" which is
> roughly where the graph goes back to normal (much lower System, much higher
> User).
>
> This has happened a few times.
>
> I have tried playing with the sysctl vm.swappiness value (default of 60) by
> setting it to 30 (which it was at when the graph was collected) and now to
> 10.  I am not sure that helps.
>
> Any ideas?  Anyone else run into this before?
>
> 24 cores
> 64GB ram
> 4x2TB sata3 hdd
>
> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb heap) on
> this machine.
>
> 24 map slots (1gb heap each), no reducers.
>
> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.
+
Robert Dyer 2012-12-11, 03:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB