Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> MR on HBase - java.io.IOException: Pass a Delete or a Put


Copy link to this message
-
Re: MR on HBase - java.io.IOException: Pass a Delete or a Put
Yes - this was also the reason in the thread I had linked earlier.
The reason is that there are two versions of the Reducer. One in the
mapred package (that uses Iterator) and the new one in mapreduce
package (that uses Iterable). It is likely that you used the new one
with the old reduce signature.

http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapred/Reducer.html#reduce(K2,
java.util.Iterator, org.apache.hadoop.mapred.OutputCollector,
org.apache.hadoop.mapred.Reporter)
http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapreduce/Reducer.html#reduce(KEYIN,
java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer.Context)

Hope that clarifies.
--Suraj

On Sat, Jul 30, 2011 at 7:58 AM, Narayanan K <[EMAIL PROTECTED]> wrote:
> Hi
>
> Many Thanks to all who responded!!
>
> I sat the whole day to solve this and to my delight, I finally found the
> issue & pinpointed where this exception kept on recurring.
>
> Though my explanation may not make sense, but this is what I changed in my
> existing code to get the reducer running fine:
> (I just changed *Iterator* to *Iterable*.)
>
>
> *Erronious reduce format
> *
> public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
> IntArrayWritable, ImmutableBytesWritable> {
>
>        public void reduce(ImmutableBytesWritable key, Iterator
> <IntArrayWritable> values, Context context)
>                throws IOException, InterruptedException {
>
>        while (values.hasNext())
>        {
>          ....
>        }
>            Put put = new Put(key.get());
>            put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
>            context.write(key, put);
>        }
> }
>
> *Corrected/Working fine reduce format*
>
> public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
> IntArrayWritable, ImmutableBytesWritable> {
>
>        public void reduce(ImmutableBytesWritable key,
> *Iterable*<IntArrayWritable>values,
> Context context)
>                throws IOException, InterruptedException {
>
>                 * for (IntArrayWritable temp : values)
>                  {*
>                     ....
>                  }
>
>
>                 Put put = new Put(key.get());
>                 put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
>                 context.write(key, put);
>        }
> }
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Please let me know if this could be the real reason? If so, why it doesnt
> support Iterator for values in reduce?
>
> Thanks,
> Narayanan
>
> On Fri, Jul 29, 2011 at 10:02 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>> Study the mapreduce examples in unit tests or under our mapreduce
>> package.  Below looks fine to me.  Maybe its how the job is
>> configured.
>>
>> St.Ack
>> P.S. you don't have to find our src in random locations; e.g. our paul
>> smiths' apache home dir.  Our src is here: hbase.org
>>
>> St.Ack
>>
>> On Fri, Jul 29, 2011 at 8:53 AM, Narayanan K <[EMAIL PROTECTED]>
>> wrote:
>> > Hi Stack/Suraj,
>> >
>> > I tried my MR code on HBase to 0.90.x version.
>> >
>> > But I am getting the same exception after the Map stage is complete:
>> > *java.io.IOException:
>> > Pass a Delete or a Put*.
>> >
>> > I did a search on web and found the *TableOutputFormat* source code where
>> > the *write* method is throwing the "*Pass a Delete or Put*" error:
>> >
>> http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
>> > (Line 92-96)
>> >
>> > From this, I understand that the value in Reduce Output key,value pair
>> > should be an instance of Put or Delete.
>> >
>> > My Reduce is also throwing an instance of Put :
>> >
>> > public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
>> > Text>
>> > {