Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> where distributed cache start working


Copy link to this message
-
Re: where distributed cache start working
Do you debug it using LocalJobRunner ? In local mode, TaskRunner won't
been called.
In local mode, mapper task runs in thread rather than forked jvm. The
TaskRunner only been called in distributed mode.
2010/8/22 Gang Luo <[EMAIL PROTECTED]>:
> Thanks Jeff.
>
> However, are you sure TaskRunner.run() is also used in the new API? I use btrace
> to trace the function call but didn't find this function had been called
> anywhere.
>
>
> One more question about distributed cache. After I call
> DistributedCache.purgeCache, I think the local cached files should be deleted or
> invalidated. However ,When I run the same job with the purge operation at the
> end multiple times, I find the local files have never been deleted and the
> modification time is when the first job run. How can I ask my job to
> re-distributed the cache again anyway?
>
> Thanks,
> -Gang
>
>
>
>
> ----- 原始邮件 ----
> 发件人: Jeff Zhang <[EMAIL PROTECTED]>
> 收件人: [EMAIL PROTECTED]
> 发送日期: 2010/8/20 (周��) 11:22:49 上午
> 主   题: Re: where distributed cache start working
>
> Hi Gang,
>
> In the TaskRunner's run() method, hadoop will download the cache files
> which you set on the client side to local, then the forked child jvm
> can use these cache files locally.
>
>
>
> On Fri, Aug 20, 2010 at 8:08 AM, Gang Luo <[EMAIL PROTECTED]> wrote:
>> Hi all,
>> I go through the code, but couldn't find the place where distributed cache
>>start
>> working. I want to know between DistriubtedCache.addCacheFile at the master
>>node
>> and DistributedCache.getLocalCacheFiles at the client side, when and where are
>> the files get distributed.
>>
>>
>> Thanks,
>> -Gang
>>
>>
>>
>>
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>
>
>

--
Best Regards

Jeff Zhang
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB