Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Predicting how many values will I see in a call to reduce?


+
Anthony Urso 2010-11-07, 13:38
+
Niels Basjes 2010-11-08, 11:32
Copy link to this message
-
Re: Predicting how many values will I see in a call to reduce?
Lance Norskog 2010-11-09, 07:40
It is key to the scheduling paradigm of Hadoop that it doesn't have to
tell you how many or when. It would have to store up all of the data
for your key before activating your reducer. This is exactly what it
cannot do and scale.

(right?)

On Mon, Nov 8, 2010 at 3:32 AM, Niels Basjes <[EMAIL PROTECTED]> wrote:
> Hi,
>
> 2010/11/7 Anthony Urso <[EMAIL PROTECTED]>
>>
>> Is there any way to know how many values I will see in a call to
>> reduce without first counting through them all with the iterator?
>>
>> Under 0.21? 0.20? 0.19?
>
> I've looked for an answer to the same question a while ago and came to the
> conclusion that you can't.
> The main limit is that the Iterator does not have a "size" or "length"
> method.
>
> --
> Met vriendelijke groeten,
>
> Niels Basjes
>

--
Lance Norskog
[EMAIL PROTECTED]
+
Owen OMalley 2010-11-09, 16:28