Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Extending ArrayWritable, Using Combiner and Spill Failed error


Copy link to this message
-
Re: Extending ArrayWritable, Using Combiner and Spill Failed error
Lance Norskog 2011-08-19, 05:03
Combiners are supposed to emit exactly what they receive. Hadoop
should look at what the combiner spits out and blow up.

On Thu, Aug 18, 2011 at 9:32 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Yep, as John points out, your trouble is related to what your
> reducer/combiner is emitting, from your snippet below:
>
> On Fri, Aug 19, 2011 at 2:14 AM, vipul sharma <[EMAIL PROTECTED]> wrote:
>>     @Override
>>     public void reduce(Text key, Iterable<IntArrayWritable> values, Context
>> context) throws IOException, InterruptedException {
>>            context.write(key, new Text(some_stuff));
>
> Since these emits would occur on the map side, you lose the
> ArrayWritable typing there cause your value has now become Text.
>
> --
> Harsh J
>

--
Lance Norskog
[EMAIL PROTECTED]