HDFS, mail # user - What is the preferred way to pass a small number of configuration parameters to a mapper or reducer

What is the preferred way to pass a small number of configuration parameters to a mapper or reducer

Kshiva Kps 2012-12-28, 08:17

Which one is current ..
What is the preferred way to pass a small number of configuration
parameters to a mapper or reducer?

*A. *As key-value pairs in the jobconf object.

* *

*B. *As a custom input key-value pair passed to each mapper or reducer.

* *

*C. *Using a plain text file via the Distributedcache, which each mapper
or reducer reads.

* *

*D. *Through a static variable in the MapReduce driver class (i.e., the
class that submits the MapReduce job).

*Answer: B*

Re: What is the preferred way to pass a small number of configuration parameters to a mapper or reducer

Ted Dunning 2012-12-28, 21:10

Answer B sounds pathologically bad to me.

A or C are the only viable options.

Neither B nor D work. B fails because it would be extremely hard to get
the right records to the right components and because it pollutes data
input with configuration data. D fails because statics don't work in
parallel programs.
On Fri, Dec 28, 2012 at 12:17 AM, Kshiva Kps <[EMAIL PROTECTED]> wrote:

>
> Which one is current ..
>
>
> What is the preferred way to pass a small number of configuration
> parameters to a mapper or reducer?
>
>
>
>
>
> *A. *As key-value pairs in the jobconf object.
>
> * *
>
> *B. *As a custom input key-value pair passed to each mapper or reducer.
>
> * *
>
> *C. *Using a plain text file via the Distributedcache, which each mapper
> or reducer reads.
>
> * *
>
> *D. *Through a static variable in the MapReduce driver class (i.e., the
> class that submits the MapReduce job).
>
>
>
> *Answer: B*
>
>
>

Re: What is the preferred way to pass a small number of configuration parameters to a mapper or reducer

Michel Segel 2012-12-28, 22:19

Sounds like someone is cheating on a test...

Sent from a remote device. Please excuse any typos...

Mike Segel

On Dec 28, 2012, at 3:10 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Answer B sounds pathologically bad to me.
>
> A or C are the only viable options.
>
> Neither B nor D work. B fails because it would be extremely hard to get the right records to the right components and because it pollutes data input with configuration data. D fails because statics don't work in parallel programs.
>
>
> On Fri, Dec 28, 2012 at 12:17 AM, Kshiva Kps <[EMAIL PROTECTED]> wrote:
>>
>> Which one is current ..
>>
>> What is the preferred way to pass a small number of configuration parameters to a mapper or reducer?
>>
>>
>> A. As key-value pairs in the jobconf object.
>>
>> B. As a custom input key-value pair passed to each mapper or reducer.
>>
>> C. Using a plain text file via the Distributedcache, which each mapper or reducer reads.
>>
>> D. Through a static variable in the MapReduce driver class (i.e., the class that submits the MapReduce job).
>>
>> Answer: B
>

Re: What is the preferred way to pass a small number of configuration parameters to a mapper or reducer

Edward Capriolo 2012-12-29, 01:20

Yes. another big data, data scientist, no ops, devops, cloud computing
specialist is born. Thank goodness we have multiple choice tests to
identify the best coders and administrators.

On Friday, December 28, 2012, Michel Segel <[EMAIL PROTECTED]>
wrote:
> Sounds like someone is cheating on a test...
>
> Sent from a remote device. Please excuse any typos...
> Mike Segel
> On Dec 28, 2012, at 3:10 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>
> Answer B sounds pathologically bad to me.
> A or C are the only viable options.
> Neither B nor D work. B fails because it would be extremely hard to get
the right records to the right components and because it pollutes data
input with configuration data. D fails because statics don't work in
parallel programs.
>
> On Fri, Dec 28, 2012 at 12:17 AM, Kshiva Kps <[EMAIL PROTECTED]> wrote:
>
> Which one is current ..
>
> What is the preferred way to pass a small number of configuration
parameters to a mapper or reducer?
>
>
>
>
>
> A. As key-value pairs in the jobconf object.
>
>
>
> B. As a custom input key-value pair passed to each mapper or reducer.
>
>

Re: What is the preferred way to pass a small number of configuration parameters to a mapper or reducer

Jay Vyas 2012-12-29, 01:29

the only way to implement B is by doing A (that i know of, at least).
Also, the word "each" is clearly a dead giveaway that B is the wrong
answer, since it implies special logic for communicating with individual
mappers/reducers.

On Fri, Dec 28, 2012 at 8:20 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> Yes. another big data, data scientist, no ops, devops, cloud computing
> specialist is born. Thank goodness we have multiple choice tests to
> identify the best coders and administrators.
>

--
Jay Vyas
http://jayunit100.blogspot.com