Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> DistributedCache in NewAPI on 0.20.X branch


Copy link to this message
-
Re: DistributedCache in NewAPI on 0.20.X branch
Hi Shi
         Try out the following,it could get things working
 Use DistributedCache.getCacheFiles() instead of
DistributedCache.getLocalCacheFiles()

public void setup(JobConf job)
{
            DistributedCache.getLocalCacheFiles(job)
.
.
.
}

If that also doesn't seem to work and if you have the file name in handy
which you populated on cache, use it the normal java way

job submission:
hadoop jar ............    -files /home/x/y/testFile.txt <input> <output>

in your setup() method, just retrieve the files like
public void setup(JobConf job)
{
       File file = new File("testFile.txt");
}
Regards
Bejoy.K.S

On Fri, Dec 16, 2011 at 4:09 AM, Shi Yu <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am using 0.20.X branch.  However, I need to use the new API because it
> has the cleanup(context) method in Mapper.  However, I am confused about
> how to load the cached files in mapper.   I could load the DistributedCache
> files using old API (JobConf),  but in new API it always returns null.   I
> read some previous discussions that "on 0.20.X branch, calling
> DistributedCache using old API is encouraged."  My question is:  Is it
> possible to use DistributedCache using new API, or the only possible way is
> to upgrade to higher version?    Or I use a wrong method in my code?
>  Thanks!
>
>
> The relevant code is something as follows:
>
> class Map extends Mapper <K1, V1, K2, V2>{
>
> public void setup(Mapper<K1,V1, K2, V2>.Context context) throws
> IOException{
>  try{
>   Path[] localFiles;
>   localFiles = DistributedCache.**getLocalCacheFiles(context.**
> getConfiguration());
>   String file1 = localFiles[0].toUri.getPath();   // return null?
>    ....
>
>  }
>
>
> }
>
> }
>
> public int run(String[] args) throws IOException, URISyntaxException{
>   ...
>  //set the configuration
>
>  Job job = new Job(conf,"Myjob");
>  DistributeCache.addCacheFile(**new URI(hdfs://server:9000/file1),
> job.getConfiguration);
>
> }
>
> BR,
>
>
> Shi
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB