Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Rapid growth in Non DFS Used disk space

Kester, Scott 2011-05-13, 17:40
Todd Lipcon 2011-05-13, 17:48
Copy link to this message
Re: Rapid growth in Non DFS Used disk space

On May 13, 2011, at 10:48 AM, Todd Lipcon wrote:
>> 2) Any ideas on what is driving the growth in Non DFS Used space?   I
>> looked for things like growing log files on the datanodes but didn't find
>> anything.
> Logs are one possible culprit. Another is to look for old files that might
> be orphaned in your mapred.local.dir - there have been bugs in the past
> where we've leaked files. If you shut down the TaskTrackers, you can safely
> delete everything from within mapred.local.dirs.

Part of our S.O.P. during Hadoop bounces is to wipe mapred.local out.  The TT doesn't properly clean up after itself.
Kester, Scott 2011-05-13, 20:41
suresh srinivas 2011-05-15, 04:20
Kester, Scott 2011-05-16, 15:50