Has anyone experienced a TaskTracker/DataNode behaving like the attached
image?

This was during a MR job (which runs often).  Note the extremely high
System CPU time.  Upon investigating I saw that out of 64GB ram the system
had allocated almost 45GB to cache!

I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync" which
is roughly where the graph goes back to normal (much lower System, much
higher User).

This has happened a few times.

I have tried playing with the sysctl vm.swappiness value (default of 60) by
setting it to 30 (which it was at when the graph was collected) and now to
10.  I am not sure that helps.

Any ideas?  Anyone else run into this before?

24 cores
64GB ram
4x2TB sata3 hdd

Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb heap) on
this machine.

24 map slots (1gb heap each), no reducers.

Also running HBase 0.94.2 with a RS (8gb ram) on this machine.

Oracle frequently recommends vm.swappiness = 0 to get well behaved RAC nodes.  Otherwise you start paging out things you don't usually want paged out in favor of a larger filesystem cache.

There is also a vm parameter that controls the minimum size of the free chain, might want to increase that a bit.

Also, look into hosting your JVM heap on huge pages, they can't be paged out and will help the JVM perform better too.

On Dec 8, 2012, at 6:09 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:

> Has anyone experienced a TaskTracker/DataNode behaving like the attached image?
>
> This was during a MR job (which runs often).  Note the extremely high System CPU time.  Upon investigating I saw that out of 64GB ram the system had allocated almost 45GB to cache!
>
> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync" which is roughly where the graph goes back to normal (much lower System, much higher User).
>
> This has happened a few times.
>
> I have tried playing with the sysctl vm.swappiness value (default of 60) by setting it to 30 (which it was at when the graph was collected) and now to 10.  I am not sure that helps.
>
> Any ideas?  Anyone else run into this before?
>
> 24 cores
> 64GB ram
> 4x2TB sata3 hdd
>
> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb heap) on this machine.
>
> 24 map slots (1gb heap each), no reducers.
>
> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.
> <cpu-use.png>

Hi,

I always set  "vm.swappiness = 0" for my hadoop servers (PostgreSQL servers too).  

The reason is that Linux moves memory pages to swap space if they have not been accessed for a period of time (swapping).  Java virtual machine (JVM) does not act well in the case of swapping that will make MapReduce (and HBase and ZooKeeper) run into trouble.   So I would suggest to set  vm.swappiness = 0.

Thanks
ac

On 9 Dec 2012, at 12:58 PM, seth wrote:

> Oracle frequently recommends vm.swappiness = 0 to get well behaved RAC nodes.  Otherwise you start paging out things you don't usually want paged out in favor of a larger filesystem cache.
>
> There is also a vm parameter that controls the minimum size of the free chain, might want to increase that a bit.
>
> Also, look into hosting your JVM heap on huge pages, they can't be paged out and will help the JVM perform better too.
>
> On Dec 8, 2012, at 6:09 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:
>
>> Has anyone experienced a TaskTracker/DataNode behaving like the attached image?
>>
>> This was during a MR job (which runs often).  Note the extremely high System CPU time.  Upon investigating I saw that out of 64GB ram the system had allocated almost 45GB to cache!
>>
>> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync" which is roughly where the graph goes back to normal (much lower System, much higher User).
>>
>> This has happened a few times.
>>
>> I have tried playing with the sysctl vm.swappiness value (default of 60) by setting it to 30 (which it was at when the graph was collected) and now to 10.  I am not sure that helps.
>>
>> Any ideas?  Anyone else run into this before?
>>
>> 24 cores
>> 64GB ram
>> 4x2TB sata3 hdd
>>
>> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb heap) on this machine.
>>
>> 24 map slots (1gb heap each), no reducers.
>>
>> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.
>> <cpu-use.png>

Are you sure that 24 map slots is a good number for this machine?
Remember that you have three services (DN, TT and HRegionServer) with
with a 12 GB for Heap.
Try to use a lower number of map slots (12 for example) and launch your
MR job again.
Can you share your logs in pastebin?
On Sat 08 Dec 2012 07:09:02 PM CST, Robert Dyer wrote:
> Has anyone experienced a TaskTracker/DataNode behaving like the
> attached image?
>
> This was during a MR job (which runs often).  Note the extremely high
> System CPU time.  Upon investigating I saw that out of 64GB ram the
> system had allocated almost 45GB to cache!
>
> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync"
> which is roughly where the graph goes back to normal (much lower
> System, much higher User).
>
> This has happened a few times.
>
> I have tried playing with the sysctl vm.swappiness value (default of
> 60) by setting it to 30 (which it was at when the graph was collected)
> and now to 10.  I am not sure that helps.
>
> Any ideas?  Anyone else run into this before?
>
> 24 cores
> 64GB ram
> 4x2TB sata3 hdd
>
> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb
> heap) on this machine.
>
> 24 map slots (1gb heap each), no reducers.
>
> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.

--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

Yes but even with a MR running, it is only 36GB heap total out of 64GB
ram.  This leaves plenty for OS and caching.

The problem seems to be the OS preferring to cache over giving space to the
applications.  Once I drop the caches and rerun the MR job again several
times, it runs perfectly fine.
On Dec 8, 2012 7:06 PM, "Marcos Ortiz" <[EMAIL PROTECTED]> wrote:

> Are you sure that 24 map slots is a good number for this machine?
> Remember that you have three services (DN, TT and HRegionServer) with
> with a 12 GB for Heap.
> Try to use a lower number of map slots (12 for example) and launch your
> MR job again.
> Can you share your logs in pastebin?
>
>
> On Sat 08 Dec 2012 07:09:02 PM CST, Robert Dyer wrote:
>
>> Has anyone experienced a TaskTracker/DataNode behaving like the
>> attached image?
>>
>> This was during a MR job (which runs often).  Note the extremely high
>> System CPU time.  Upon investigating I saw that out of 64GB ram the
>> system had allocated almost 45GB to cache!
>>
>> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync"
>> which is roughly where the graph goes back to normal (much lower
>> System, much higher User).
>>
>> This has happened a few times.
>>
>> I have tried playing with the sysctl vm.swappiness value (default of
>> 60) by setting it to 30 (which it was at when the graph was collected)
>> and now to 10.  I am not sure that helps.
>>
>> Any ideas?  Anyone else run into this before?
>>
>> 24 cores
>> 64GB ram
>> 4x2TB sata3 hdd
>>
>> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb
>> heap) on this machine.
>>
>> 24 map slots (1gb heap each), no reducers.
>>
>> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.
>>
>
> --
> Marcos Luis Ortíz Valmaseda
> about.me/marcosortiz <http://about.me/marcosortiz>
> @marcosluis2186 <http://twitter.com/**marcosluis2186<http://twitter.com/marcosluis2186>
> >
>
>
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
> INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/**universidad.uci<http://www.facebook.com/universidad.uci>
> http://www.flickr.com/photos/**universidad_uci<http://www.flickr.com/photos/universidad_uci>
>

You may want to check why System time is high. Check your system call stats. This should give you some clue.

-Bharath
________________________________
 From: Robert Dyer <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Bharath Mundlapudi <[EMAIL PROTECTED]>
Sent: Monday, December 10, 2012 7:32 PM
Subject: Re: Strange machine behavior
 

Yes there is performance impact.  It should be visible from the graph I attached.  Basically, the CPU is spending much more time on System and the User time is lowered.

When this happens (if I don't do a drop_caches in time) the MR job winds up taking significantly longer than usual.

On Mon, Dec 10, 2012 at 8:06 PM, Bharath Mundlapudi <[EMAIL PROTECTED]> wrote:

Are you seeing any performance impact with this cache increase? It is normal in linux system to grab high cache level.
>
>
>
>-Bharath
>
>
>
>________________________________
> From: Andy Isaacson <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Monday, December 10, 2012 11:23 AM
>Subject: Re: Strange machine behavior
>
>
>What kernel did you see this on? Was there significant swap traffic
>(si/so in vmstat output) during the high-system-time period?
>
>BTW, you don't need to nor do you want to run sync(1) when
>manipulating drop_caches, it just causes additional noise and
>slowdown. drop_caches doesn't have any impact on correctness; it won't
>cause data loss (by dropping a dirty page or whatever). I've had sync
>calls take 10 minutes to complete, so the unnecessary impact can be
>significant.
>
>-andy
>
>On Sat, Dec 8, 2012 at 4:09 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:
>> Has anyone experienced a TaskTracker/DataNode behaving like the attached
>> image?
>>
>> This was during a MR job (which runs often).  Note the extremely high System
>> CPU time.  Upon investigating I saw that out of 64GB ram the system had
>> allocated
 almost 45GB to cache!
>>
>> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync" which is
>> roughly where the graph goes back to normal (much lower System, much higher
>> User).
>>
>> This has happened a few times.
>>
>> I have tried playing with the sysctl vm.swappiness value (default of 60) by
>> setting it to 30 (which it was at when the graph was collected) and now to
>> 10.  I am not sure that helps.
>>
>> Any ideas?  Anyone else run into this before?
>>
>> 24 cores
>> 64GB ram
>> 4x2TB sata3 hdd
>>
>> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb heap) on
>> this machine.
>>
>> 24 map slots (1gb heap each), no reducers.
>>
>> Also running HBase 0.94.2 with a RS (8gb ram) on this machine.
>
>
>
--

Robert Dyer
[EMAIL PROTECTED]