Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - where distributed cache start working


Copy link to this message
-
Re: where distributed cache start working
Jeff Zhang 2010-08-23, 04:47
Do you debug it using LocalJobRunner ? In local mode, TaskRunner won't
been called.
In local mode, mapper task runs in thread rather than forked jvm. The
TaskRunner only been called in distributed mode.
2010/8/22 Gang Luo <[EMAIL PROTECTED]>:
> Thanks Jeff.
>
> However, are you sure TaskRunner.run() is also used in the new API? I use btrace
> to trace the function call but didn't find this function had been called
> anywhere.
>
>
> One more question about distributed cache. After I call
> DistributedCache.purgeCache, I think the local cached files should be deleted or
> invalidated. However ,When I run the same job with the purge operation at the
> end multiple times, I find the local files have never been deleted and the
> modification time is when the first job run. How can I ask my job to
> re-distributed the cache again anyway?
>
> Thanks,
> -Gang
>
>
>
>
> ----- 原始邮件 ----
> 发件人: Jeff Zhang <[EMAIL PROTECTED]>
> 收件人: [EMAIL PROTECTED]
> 发送日期: 2010/8/20 (周��) 11:22:49 上午
> 主   题: Re: where distributed cache start working
>
> Hi Gang,
>
> In the TaskRunner's run() method, hadoop will download the cache files
> which you set on the client side to local, then the forked child jvm
> can use these cache files locally.
>
>
>
> On Fri, Aug 20, 2010 at 8:08 AM, Gang Luo <[EMAIL PROTECTED]> wrote:
>> Hi all,
>> I go through the code, but couldn't find the place where distributed cache
>>start
>> working. I want to know between DistriubtedCache.addCacheFile at the master
>>node
>> and DistributedCache.getLocalCacheFiles at the client side, when and where are
>> the files get distributed.
>>
>>
>> Thanks,
>> -Gang
>>
>>
>>
>>
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>
>
>

--
Best Regards

Jeff Zhang