Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Extending ArrayWritable, Using Combiner and Spill Failed error


+
vipul sharma 2011-08-18, 20:44
+
John Armstrong 2011-08-18, 20:51
+
Harsh J 2011-08-19, 04:32
Copy link to this message
-
Re: Extending ArrayWritable, Using Combiner and Spill Failed error
Combiners are supposed to emit exactly what they receive. Hadoop
should look at what the combiner spits out and blow up.

On Thu, Aug 18, 2011 at 9:32 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Yep, as John points out, your trouble is related to what your
> reducer/combiner is emitting, from your snippet below:
>
> On Fri, Aug 19, 2011 at 2:14 AM, vipul sharma <[EMAIL PROTECTED]> wrote:
>>     @Override
>>     public void reduce(Text key, Iterable<IntArrayWritable> values, Context
>> context) throws IOException, InterruptedException {
>>            context.write(key, new Text(some_stuff));
>
> Since these emits would occur on the map side, you lose the
> ArrayWritable typing there cause your value has now become Text.
>
> --
> Harsh J
>

--
Lance Norskog
[EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB