Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: Auto clean DistCache?


+
Abdelrhman Shettia 2013-03-26, 19:53
+
Vinod Kumar Vavilapalli 2013-03-26, 20:44
+
Abdelrahman Shettia 2013-03-26, 23:12
+
Jean-Marc Spaggiari 2013-03-27, 01:00
+
Koji Noguchi 2013-03-27, 13:21
+
Jean-Marc Spaggiari 2013-03-27, 13:37
+
Harsh J 2013-03-28, 06:33
+
Jean-Marc Spaggiari 2013-03-28, 16:02
Copy link to this message
-
Re: Auto clean DistCache?
Vinod Kumar Vavilapalli 2013-03-26, 20:43

You can control the limit of these cache files, the default is 10GB (value of 10737418240L): Try changing local.cache.size or mapreduce.tasktracker.cache.local.size in mapred-site.xml

Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Mar 25, 2013, at 5:16 PM, Jean-Marc Spaggiari wrote:

> Hi,
>
> Each time my MR job is run, a directory is created on the TaskTracker
> under mapred/local/taskTracker/hadoop/distcache (based on my
> configuration).
>
> I looked at the directory today, and it's hosting thousands of
> directories and more than 8GB of data there.
>
> Is there a way to automatically delete this directory when the job is done?
>
> Thanks,
>
> JM