Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: JobCache directory cleanup


Copy link to this message
-
Re: JobCache directory cleanup
Thanks for replies!

keep.failed.task.files set to false.
Config of one of the jobs attached.
On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <[EMAIL PROTECTED]
> wrote:

> Good point. Forgot that one :-)
>
>
> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
> [EMAIL PROTECTED]> wrote:
>
>>
>>
>> Can you check the job configuration for these ~100 jobs? Do they have
>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>> it doesn't, it could be a bug.
>>
>> Sharing your configs for these jobs will definitely help.
>>
>> Thanks,
>> +Vinod
>>
>>
>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>
>
--
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
[EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB