Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: JobCache directory cleanup


Copy link to this message
-
Re: JobCache directory cleanup
Ivan Tretyakov 2013-01-11, 09:58
Thanks for replies!

keep.failed.task.files set to false.
Config of one of the jobs attached.
On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <[EMAIL PROTECTED]
> wrote:

> Good point. Forgot that one :-)
>
>
> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
> [EMAIL PROTECTED]> wrote:
>
>>
>>
>> Can you check the job configuration for these ~100 jobs? Do they have
>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>> it doesn't, it could be a bug.
>>
>> Sharing your configs for these jobs will definitely help.
>>
>> Thanks,
>> +Vinod
>>
>>
>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>
>
--
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
[EMAIL PROTECTED]