Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> What the parameter "mapred.indirect.input.file" means?


+
Pedro Sá da Costa 2013-03-19, 10:43
Copy link to this message
-
Re: What the parameter "mapred.indirect.input.file" means?
>From src/test/org/apache/hadoop/mapred/GenericMRLoadGenerator.java, looks
like it is used to generate IndirectSplit's:

    public InputSplit[] getSplits(JobConf job, int numSplits)
        throws IOException {

      Path src = new Path(job.get("mapred.indirect.input.file", null));
      FileSystem fs = src.getFileSystem(job);

      ArrayList<IndirectSplit> splits = new
ArrayList<IndirectSplit>(numSplits);
      LongWritable key = new LongWritable();
      Text value = new Text();
      for (SequenceFile.Reader sl = new SequenceFile.Reader(fs, src, job);
           sl.next(key, value);) {
        splits.add(new IndirectSplit(new Path(value.toString()),
key.get()));
      }

      return splits.toArray(new IndirectSplit[splits.size()]);

On Tue, Mar 19, 2013 at 3:43 AM, Pedro Sá da Costa <[EMAIL PROTECTED]>wrote:

> Hi,
>
> What the parameter "mapred.indirect.input.file" means? For what it is used?
>
> --
> Best regards,
> P
>