-Re: JobCache directory cleanup
Ivan Tretyakov 2013-01-11, 09:58
Thanks for replies!
keep.failed.task.files set to false.
Config of one of the jobs attached.
On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <[EMAIL PROTECTED]
> Good point. Forgot that one :-)
> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
> [EMAIL PROTECTED]> wrote:
>> Can you check the job configuration for these ~100 jobs? Do they have
>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>> it doesn't, it could be a bug.
>> Sharing your configs for these jobs will definitely help.
>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>> [EMAIL PROTECTED]> wrote:
>>> I've found that jobcache directory became very large on our cluster,
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G /data1/mapred/local/taskTracker/user/jobcache
>>> 464G /data2/mapred/local/taskTracker/user/jobcache
>>> 454G /data3/mapred/local/taskTracker/user/jobcache
>>> And it stores information for about 100 jobs:
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/ | sort | uniq
>>> | wc -l
+7 812 640 38 76