Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> What the parameter "mapred.indirect.input.file" means?


Copy link to this message
-
Re: What the parameter "mapred.indirect.input.file" means?
>From src/test/org/apache/hadoop/mapred/GenericMRLoadGenerator.java, looks
like it is used to generate IndirectSplit's:

    public InputSplit[] getSplits(JobConf job, int numSplits)
        throws IOException {

      Path src = new Path(job.get("mapred.indirect.input.file", null));
      FileSystem fs = src.getFileSystem(job);

      ArrayList<IndirectSplit> splits = new
ArrayList<IndirectSplit>(numSplits);
      LongWritable key = new LongWritable();
      Text value = new Text();
      for (SequenceFile.Reader sl = new SequenceFile.Reader(fs, src, job);
           sl.next(key, value);) {
        splits.add(new IndirectSplit(new Path(value.toString()),
key.get()));
      }

      return splits.toArray(new IndirectSplit[splits.size()]);

On Tue, Mar 19, 2013 at 3:43 AM, Pedro Sá da Costa <[EMAIL PROTECTED]>wrote:

> Hi,
>
> What the parameter "mapred.indirect.input.file" means? For what it is used?
>
> --
> Best regards,
> P
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB