Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - DistributedCache in NewAPI on 0.20.X branch


Copy link to this message
-
Re: DistributedCache in NewAPI on 0.20.X branch
Bejoy Ks 2011-12-16, 09:33
Hi Shi
         Try out the following,it could get things working
 Use DistributedCache.getCacheFiles() instead of
DistributedCache.getLocalCacheFiles()

public void setup(JobConf job)
{
            DistributedCache.getLocalCacheFiles(job)
.
.
.
}

If that also doesn't seem to work and if you have the file name in handy
which you populated on cache, use it the normal java way

job submission:
hadoop jar ............    -files /home/x/y/testFile.txt <input> <output>

in your setup() method, just retrieve the files like
public void setup(JobConf job)
{
       File file = new File("testFile.txt");
}
Regards
Bejoy.K.S

On Fri, Dec 16, 2011 at 4:09 AM, Shi Yu <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am using 0.20.X branch.  However, I need to use the new API because it
> has the cleanup(context) method in Mapper.  However, I am confused about
> how to load the cached files in mapper.   I could load the DistributedCache
> files using old API (JobConf),  but in new API it always returns null.   I
> read some previous discussions that "on 0.20.X branch, calling
> DistributedCache using old API is encouraged."  My question is:  Is it
> possible to use DistributedCache using new API, or the only possible way is
> to upgrade to higher version?    Or I use a wrong method in my code?
>  Thanks!
>
>
> The relevant code is something as follows:
>
> class Map extends Mapper <K1, V1, K2, V2>{
>
> public void setup(Mapper<K1,V1, K2, V2>.Context context) throws
> IOException{
>  try{
>   Path[] localFiles;
>   localFiles = DistributedCache.**getLocalCacheFiles(context.**
> getConfiguration());
>   String file1 = localFiles[0].toUri.getPath();   // return null?
>    ....
>
>  }
>
>
> }
>
> }
>
> public int run(String[] args) throws IOException, URISyntaxException{
>   ...
>  //set the configuration
>
>  Job job = new Job(conf,"Myjob");
>  DistributeCache.addCacheFile(**new URI(hdfs://server:9000/file1),
> job.getConfiguration);
>
> }
>
> BR,
>
>
> Shi
>
>