Hi,

I am new to hadoop. I have a set of files and I want to assign each file to
a mapper. Also in mapper there should be a way to know the complete path of
the file. Can you please tell me how to do that ?

Thanks,
Govind

--
Govind Kothari
Graduate Student
Dept. of Computer Science
University of Maryland College Park

<---Seek Excellence, Success will Follow --->

Hi Govind,

You should use overwrite your FileInputFormat isSplitable function in a
class say myFileInputFormat extends FileInputFormat as follows:

@Override
    public boolean isSplitable(FileSystem fs, Path filename){
        return false;
    }

Then one you use your myFileInputFormat class. To know the path, write the
following in your mapper class:

@Override
    public void configure(JobConf job) {

        Path inputPath = new Path(job.get("map.input.file"));

}

~cheers,

Mark

On Tue, Jul 5, 2011 at 1:04 PM, Govind Kothari <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am new to hadoop. I have a set of files and I want to assign each file to
> a mapper. Also in mapper there should be a way to know the complete path of
> the file. Can you please tell me how to do that ?
>
> Thanks,
> Govind
>
> --
> Govind Kothari
> Graduate Student
> Dept. of Computer Science
> University of Maryland College Park
>
> <---Seek Excellence, Success will Follow --->
>

I've done this before by placing the name of each file to process into a single file (newline separated) and using the NLineInputFormat class as the input format. Run your job with the single file with all of the file names to process as the input. Each mapper will then be handed one line (this is tunable) from the single input file. The line will contain the name of the file to process.

You can also write your own InputFormat class that creates a split for each file.

Both of these options have scalability issues which begs the question: why one file per mapper?

-----Original Message-----
From: Govind Kothari [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, July 05, 2011 3:04 PM
To: [EMAIL PROTECTED]
Subject: One file per mapper

Hi,

I am new to hadoop. I have a set of files and I want to assign each file to a mapper. Also in mapper there should be a way to know the complete path of the file. Can you please tell me how to do that ?

Thanks,
Govind

--
Govind Kothari
Graduate Student
Dept. of Computer Science
University of Maryland College Park

<---Seek Excellence, Success will Follow --->

On Tue, Jul 5, 2011 at 5:28 PM, Jim Falgout <[EMAIL PROTECTED]>wrote:

> I've done this before by placing the name of each file to process into a
> single file (newline separated) and using the NLineInputFormat class as the
> input format. Run your job with the single file with all of the file names
> to process as the input. Each mapper will then be handed one line (this is
> tunable) from the single input file. The line will contain the name of the
> file to process.
>
> You can also write your own InputFormat class that creates a split for each
> file.
>
> Both of these options have scalability issues which begs the question: why
> one file per mapper?
>
> -----Original Message-----
> From: Govind Kothari [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, July 05, 2011 3:04 PM
> To: [EMAIL PROTECTED]
> Subject: One file per mapper
>
> Hi,
>
> I am new to hadoop. I have a set of files and I want to assign each file to
> a mapper. Also in mapper there should be a way to know the complete path of
> the file. Can you please tell me how to do that ?
>
> Thanks,
> Govind
>
> --
> Govind Kothari
> Graduate Student
> Dept. of Computer Science
> University of Maryland College Park
>
> <---Seek Excellence, Success will Follow --->
>
>
You can also do this with MultipleInputs and MultipleOutputs classes. Each
source file can have a different mapper.

Actually I tried this way of putting the paths of each file in the single
file and providing it as input. But the problem is mapper is not able to
identify the path and giving an error of file not found. I am not sure why
is this so.

--
View this message in context: http://lucene.472066.n3.nabble.com/One-file-per-mapper-tp3142514p3243038.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Hi Terry

If you are having files smaller than hdfs block size and if you are using
Default TextInputFormat with the default properties for split sizes there
would be just one file per mapper.

If you are having larger file sizes, greater than the size of a hdfs block.
Please take a look at a sample implemention of 'WholeFileInputFormat' from
'Hadoop - The Definitive Guide' by Tom White.
http://books.google.co.in/books?id=Nff49D7vnJcC&pg=PA206&lpg=PA206&dq=wholefileinputformat&source=bl&ots=IifzWlbwQs&sig=9CDmS45S8pGDOaCYl6xGXnyDFE8&hl=en&sa=X&ei=VeJyUKfEE4rMrQe654G4DA&ved=0CCsQ6AEwAg#v=onepage&q=wholefileinputformat&f=false

On Mon, Oct 8, 2012 at 7:51 PM, Terry Healy <[EMAIL PROTECTED]> wrote:

> Hello-
>
> I know that it is contrary to normal Hadoop operation, but how can I
> configure my M/R job to send one complete file to each mapper task? This
> is intended to be used on many files in the 1.5 MB range as the first
> step in a chain of processes.
>
> thanks.
>