Hi All,
I am using hadoop-0.19.2 and running a Mapper only Job on cluster. It's
input path has >1000 files of 100-200MB. Since, it is Mapper only job, I
gave number Of reducer=0. So, it is using 2 mapper to run all the input
files. If we did not state the number of mapper, would n't it pick the 1
mapper per input file? Or Does the default won't it pick a fair num of
mapper according to number input file?
Thanks,
tamil

Hi Tamil,

I'd recommend upgrading to a newer release as 0.19.2 is very old. As for
your question, most input formats should set the number mappers correctly.
What input format are you using? Where did you see the number of tasks it
assigned to the job?

-Joey

On Thu, Feb 16, 2012 at 1:40 AM, Thamizhannal Paramasivam <
[EMAIL PROTECTED]> wrote:

> Hi All,
> I am using hadoop-0.19.2 and running a Mapper only Job on cluster. It's
> input path has >1000 files of 100-200MB. Since, it is Mapper only job, I
> gave number Of reducer=0. So, it is using 2 mapper to run all the input
> files. If we did not state the number of mapper, would n't it pick the 1
> mapper per input file? Or Does the default won't it pick a fair num of
> mapper according to number input file?
> Thanks,
> tamil
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Here are the input format for mapper.
Input Format: MultiFileInputFormat
MapperOutputKey : Text
MapperOutputValue: CustomWritable

I shall not be in the position to upgrade hadoop-0.19.2 for some reason.

I have checked in number of mapper on job-tracker.

Thanks,
Thamizh

On Thu, Feb 16, 2012 at 6:56 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:

> Hi Tamil,
>
> I'd recommend upgrading to a newer release as 0.19.2 is very old. As for
> your question, most input formats should set the number mappers correctly.
> What input format are you using? Where did you see the number of tasks it
> assigned to the job?
>
> -Joey
>
>
> On Thu, Feb 16, 2012 at 1:40 AM, Thamizhannal Paramasivam <
> [EMAIL PROTECTED]> wrote:
>
>> Hi All,
>> I am using hadoop-0.19.2 and running a Mapper only Job on cluster. It's
>> input path has >1000 files of 100-200MB. Since, it is Mapper only job, I
>> gave number Of reducer=0. So, it is using 2 mapper to run all the input
>> files. If we did not state the number of mapper, would n't it pick the 1
>> mapper per input file? Or Does the default won't it pick a fair num of
>> mapper according to number input file?
>> Thanks,
>> tamil
>
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>
>

Hi Tamizh
         If your input comprises of text files then changing the input format to TextInputFormat can get things right. One mapper for each hdfs block.
Regards
Bejoy K S

From handheld, Please excuse typos.

-----Original Message-----
From: Thamizhannal Paramasivam <[EMAIL PROTECTED]>
Date: Thu, 16 Feb 2012 21:33:11
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: num of reducer

Here are the input format for mapper.
Input Format: MultiFileInputFormat
MapperOutputKey : Text
MapperOutputValue: CustomWritable

I shall not be in the position to upgrade hadoop-0.19.2 for some reason.

I have checked in number of mapper on job-tracker.

Thanks,
Thamizh

On Thu, Feb 16, 2012 at 6:56 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:

> Hi Tamil,
>
> I'd recommend upgrading to a newer release as 0.19.2 is very old. As for
> your question, most input formats should set the number mappers correctly.
> What input format are you using? Where did you see the number of tasks it
> assigned to the job?
>
> -Joey
>
>
> On Thu, Feb 16, 2012 at 1:40 AM, Thamizhannal Paramasivam <
> [EMAIL PROTECTED]> wrote:
>
>> Hi All,
>> I am using hadoop-0.19.2 and running a Mapper only Job on cluster. It's
>> input path has >1000 files of 100-200MB. Since, it is Mapper only job, I
>> gave number Of reducer=0. So, it is using 2 mapper to run all the input
>> files. If we did not state the number of mapper, would n't it pick the 1
>> mapper per input file? Or Does the default won't it pick a fair num of
>> mapper according to number input file?
>> Thanks,
>> tamil
>
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>
>

Is your data size 100-200MB *total*?

If so, then this is the expected behavior for MultiFileInputFormat. As
Bejoy says, you can switch to TextInputFormat to get one mapper per block
(min one mapper per file).

-Joey

On Thu, Feb 16, 2012 at 11:03 AM, Thamizhannal Paramasivam <
[EMAIL PROTECTED]> wrote:

> Here are the input format for mapper.
> Input Format: MultiFileInputFormat
> MapperOutputKey : Text
> MapperOutputValue: CustomWritable
>
> I shall not be in the position to upgrade hadoop-0.19.2 for some reason.
>
> I have checked in number of mapper on job-tracker.
>
> Thanks,
> Thamizh
>
>
> On Thu, Feb 16, 2012 at 6:56 PM, Joey Echeverria <[EMAIL PROTECTED]>wrote:
>
>> Hi Tamil,
>>
>> I'd recommend upgrading to a newer release as 0.19.2 is very old. As for
>> your question, most input formats should set the number mappers correctly.
>> What input format are you using? Where did you see the number of tasks it
>> assigned to the job?
>>
>> -Joey
>>
>>
>> On Thu, Feb 16, 2012 at 1:40 AM, Thamizhannal Paramasivam <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi All,
>>> I am using hadoop-0.19.2 and running a Mapper only Job on cluster. It's
>>> input path has >1000 files of 100-200MB. Since, it is Mapper only job, I
>>> gave number Of reducer=0. So, it is using 2 mapper to run all the input
>>> files. If we did not state the number of mapper, would n't it pick the 1
>>> mapper per input file? Or Does the default won't it pick a fair num of
>>> mapper according to number input file?
>>> Thanks,
>>> tamil
>>
>>
>>
>>
>> --
>> Joseph Echeverria
>> Cloudera, Inc.
>> 443.305.9434
>>
>>
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Thank you so much to Joey & Bejoy for your suggestions.

The Job's input path has 1300-1400 text files and each of 100-200MB.

I thought, TextInputFormat spans single mapper per file and
MultiFileInputFormat spans less number mapper(<(1300-1400)) that processes
more many input files.

Which input format do you thing would be most appropriate in my case and
why?

Looking forward to your reply.

Thanks,
Thamizh
On Thu, Feb 16, 2012 at 10:06 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:

> Is your data size 100-200MB *total*?
>
> If so, then this is the expected behavior for MultiFileInputFormat. As
> Bejoy says, you can switch to TextInputFormat to get one mapper per block
> (min one mapper per file).
>
> -Joey
>
>
> On Thu, Feb 16, 2012 at 11:03 AM, Thamizhannal Paramasivam <
> [EMAIL PROTECTED]> wrote:
>
>> Here are the input format for mapper.
>> Input Format: MultiFileInputFormat
>> MapperOutputKey : Text
>> MapperOutputValue: CustomWritable
>>
>> I shall not be in the position to upgrade hadoop-0.19.2 for some reason.
>>
>> I have checked in number of mapper on job-tracker.
>>
>> Thanks,
>> Thamizh
>>
>>
>> On Thu, Feb 16, 2012 at 6:56 PM, Joey Echeverria <[EMAIL PROTECTED]>wrote:
>>
>>> Hi Tamil,
>>>
>>> I'd recommend upgrading to a newer release as 0.19.2 is very old. As for
>>> your question, most input formats should set the number mappers correctly.
>>> What input format are you using? Where did you see the number of tasks it
>>> assigned to the job?
>>>
>>> -Joey
>>>
>>>
>>> On Thu, Feb 16, 2012 at 1:40 AM, Thamizhannal Paramasivam <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi All,
>>>> I am using hadoop-0.19.2 and running a Mapper only Job on cluster. It's
>>>> input path has >1000 files of 100-200MB. Since, it is Mapper only job, I
>>>> gave number Of reducer=0. So, it is using 2 mapper to run all the input
>>>> files. If we did not state the number of mapper, would n't it pick the 1
>>>> mapper per input file? Or Does the default won't it pick a fair num of
>>>> mapper according to number input file?
>>>> Thanks,
>>>> tamil
>>>
>>>
>>>
>>>
>>> --
>>> Joseph Echeverria
>>> Cloudera, Inc.
>>> 443.305.9434
>>>
>>>
>>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>
>

Hi Tamizh
         MultiFileInputFormat / CombineFileInputFormat is typically used
where the input files are relatively small (typically less than a block
size). When you use these, there is some loss in data locality, as all the
splits a mapper process won't be in the same node.
       TextInputFormat spawns one mapper each for one block in default (not
one per file). Here you hold data locality pretty much compared to
MultiFileInputFormat.
      If your mapper is not very short lived and has some decent amount of
processing involved then you can go with TextInputFormat . The one
consideration you need to make is, on your specified input when this job is
running it may span a larger number of map tasks there by occupying almost
all your map task slots in your cluster. If there are other tasks to be
triggred they may have to wait for free map slots. You may need to consider
using a Scheduler for fair share of slots to other parallel jobs as well,
if any.

Regards
Bejoy.K.S
On Fri, Feb 17, 2012 at 10:26 AM, Thamizhannal Paramasivam <
[EMAIL PROTECTED]> wrote:

> Thank you so much to Joey & Bejoy for your suggestions.
>
> The Job's input path has 1300-1400 text files and each of 100-200MB.
>
> I thought, TextInputFormat spans single mapper per file and
> MultiFileInputFormat spans less number mapper(<(1300-1400)) that processes
> more many input files.
>
> Which input format do you thing would be most appropriate in my case and
> why?
>
> Looking forward to your reply.
>
> Thanks,
> Thamizh
>
>
>
> On Thu, Feb 16, 2012 at 10:06 PM, Joey Echeverria <[EMAIL PROTECTED]>wrote:
>
>> Is your data size 100-200MB *total*?
>>
>> If so, then this is the expected behavior for MultiFileInputFormat. As
>> Bejoy says, you can switch to TextInputFormat to get one mapper per block
>> (min one mapper per file).
>>
>> -Joey
>>
>>
>> On Thu, Feb 16, 2012 at 11:03 AM, Thamizhannal Paramasivam <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Here are the input format for mapper.
>>> Input Format: MultiFileInputFormat
>>> MapperOutputKey : Text
>>> MapperOutputValue: CustomWritable
>>>
>>> I shall not be in the position to upgrade hadoop-0.19.2 for some reason.
>>>
>>> I have checked in number of mapper on job-tracker.
>>>
>>> Thanks,
>>> Thamizh
>>>
>>>
>>> On Thu, Feb 16, 2012 at 6:56 PM, Joey Echeverria <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi Tamil,
>>>>
>>>> I'd recommend upgrading to a newer release as 0.19.2 is very old. As
>>>> for your question, most input formats should set the number mappers
>>>> correctly. What input format are you using? Where did you see the number of
>>>> tasks it assigned to the job?
>>>>
>>>> -Joey
>>>>
>>>>
>>>> On Thu, Feb 16, 2012 at 1:40 AM, Thamizhannal Paramasivam <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi All,
>>>>> I am using hadoop-0.19.2 and running a Mapper only Job on cluster.
>>>>> It's input path has >1000 files of 100-200MB. Since, it is Mapper only job,
>>>>> I gave number Of reducer=0. So, it is using 2 mapper to run all the input
>>>>> files. If we did not state the number of mapper, would n't it pick the 1
>>>>> mapper per input file? Or Does the default won't it pick a fair num of
>>>>> mapper according to number input file?
>>>>> Thanks,
>>>>> tamil
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Joseph Echeverria
>>>> Cloudera, Inc.
>>>> 443.305.9434
>>>>
>>>>
>>>
>>
>>
>> --
>> Joseph Echeverria
>> Cloudera, Inc.
>> 443.305.9434
>>
>>
>

It worked me. Thanks a lot Bejoy.

Thanks
Thamizh

On Fri, Feb 17, 2012 at 3:08 PM, Bejoy Ks <[EMAIL PROTECTED]> wrote:

> Hi Tamizh
>          MultiFileInputFormat / CombineFileInputFormat is typically used
> where the input files are relatively small (typically less than a block
> size). When you use these, there is some loss in data locality, as all the
> splits a mapper process won't be in the same node.
>        TextInputFormat spawns one mapper each for one block in default
> (not one per file). Here you hold data locality pretty much compared to
> MultiFileInputFormat.
>       If your mapper is not very short lived and has some decent amount of
> processing involved then you can go with TextInputFormat . The one
> consideration you need to make is, on your specified input when this job is
> running it may span a larger number of map tasks there by occupying almost
> all your map task slots in your cluster. If there are other tasks to be
> triggred they may have to wait for free map slots. You may need to consider
> using a Scheduler for fair share of slots to other parallel jobs as well,
> if any.
>
> Regards
> Bejoy.K.S
>
>
>
> On Fri, Feb 17, 2012 at 10:26 AM, Thamizhannal Paramasivam <
> [EMAIL PROTECTED]> wrote:
>
>> Thank you so much to Joey & Bejoy for your suggestions.
>>
>> The Job's input path has 1300-1400 text files and each of 100-200MB.
>>
>> I thought, TextInputFormat spans single mapper per file and
>> MultiFileInputFormat spans less number mapper(<(1300-1400)) that processes
>> more many input files.
>>
>> Which input format do you thing would be most appropriate in my case and
>> why?
>>
>> Looking forward to your reply.
>>
>> Thanks,
>> Thamizh
>>
>>
>>
>> On Thu, Feb 16, 2012 at 10:06 PM, Joey Echeverria <[EMAIL PROTECTED]>wrote:
>>
>>> Is your data size 100-200MB *total*?
>>>
>>> If so, then this is the expected behavior for MultiFileInputFormat. As
>>> Bejoy says, you can switch to TextInputFormat to get one mapper per block
>>> (min one mapper per file).
>>>
>>> -Joey
>>>
>>>
>>> On Thu, Feb 16, 2012 at 11:03 AM, Thamizhannal Paramasivam <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Here are the input format for mapper.
>>>> Input Format: MultiFileInputFormat
>>>> MapperOutputKey : Text
>>>> MapperOutputValue: CustomWritable
>>>>
>>>> I shall not be in the position to upgrade hadoop-0.19.2 for some reason.
>>>>
>>>> I have checked in number of mapper on job-tracker.
>>>>
>>>> Thanks,
>>>> Thamizh
>>>>
>>>>
>>>> On Thu, Feb 16, 2012 at 6:56 PM, Joey Echeverria <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi Tamil,
>>>>>
>>>>> I'd recommend upgrading to a newer release as 0.19.2 is very old. As
>>>>> for your question, most input formats should set the number mappers
>>>>> correctly. What input format are you using? Where did you see the number of
>>>>> tasks it assigned to the job?
>>>>>
>>>>> -Joey
>>>>>
>>>>>
>>>>> On Thu, Feb 16, 2012 at 1:40 AM, Thamizhannal Paramasivam <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Hi All,
>>>>>> I am using hadoop-0.19.2 and running a Mapper only Job on cluster.
>>>>>> It's input path has >1000 files of 100-200MB. Since, it is Mapper only job,
>>>>>> I gave number Of reducer=0. So, it is using 2 mapper to run all the input
>>>>>> files. If we did not state the number of mapper, would n't it pick the 1
>>>>>> mapper per input file? Or Does the default won't it pick a fair num of
>>>>>> mapper according to number input file?
>>>>>> Thanks,
>>>>>> tamil
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Joseph Echeverria
>>>>> Cloudera, Inc.
>>>>> 443.305.9434
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Joseph Echeverria
>>> Cloudera, Inc.
>>> 443.305.9434
>>>
>>>
>>
>