|
|
-
What the parameter "mapred.indirect.input.file" means?
Pedro Sá da Costa 2013-03-19, 10:43
Hi,
What the parameter "mapred.indirect.input.file" means? For what it is used?
-- Best regards, P
-
Re: What the parameter "mapred.indirect.input.file" means?
Ted Yu 2013-03-19, 15:06
>From src/test/org/apache/hadoop/mapred/GenericMRLoadGenerator.java, looks like it is used to generate IndirectSplit's:
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
Path src = new Path(job.get("mapred.indirect.input.file", null)); FileSystem fs = src.getFileSystem(job);
ArrayList<IndirectSplit> splits = new ArrayList<IndirectSplit>(numSplits); LongWritable key = new LongWritable(); Text value = new Text(); for (SequenceFile.Reader sl = new SequenceFile.Reader(fs, src, job); sl.next(key, value);) { splits.add(new IndirectSplit(new Path(value.toString()), key.get())); }
return splits.toArray(new IndirectSplit[splits.size()]);
On Tue, Mar 19, 2013 at 3:43 AM, Pedro Sá da Costa <[EMAIL PROTECTED]>wrote:
> Hi, > > What the parameter "mapred.indirect.input.file" means? For what it is used? > > -- > Best regards, > P >
|
|
All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by
Sematext