Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Extending ArrayWritable, Using Combiner and Spill Failed error


Copy link to this message
-
Re: Extending ArrayWritable, Using Combiner and Spill Failed error
Combiners are supposed to emit exactly what they receive. Hadoop
should look at what the combiner spits out and blow up.

On Thu, Aug 18, 2011 at 9:32 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Yep, as John points out, your trouble is related to what your
> reducer/combiner is emitting, from your snippet below:
>
> On Fri, Aug 19, 2011 at 2:14 AM, vipul sharma <[EMAIL PROTECTED]> wrote:
>>     @Override
>>     public void reduce(Text key, Iterable<IntArrayWritable> values, Context
>> context) throws IOException, InterruptedException {
>>            context.write(key, new Text(some_stuff));
>
> Since these emits would occur on the map side, you lose the
> ArrayWritable typing there cause your value has now become Text.
>
> --
> Harsh J
>

--
Lance Norskog
[EMAIL PROTECTED]