Hi Everyone,

I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink both
being HBase Tables.

On running this mapreduce code, the Map completes successfully, but before
reduce starts, I am hitting an IOException as below:

2011-07-26 23:06:07,337 WARN  [Thread-11] mapred.LocalJobRunner$Job(255):
job_local_0001
java.io.IOException: Pass a Delete or a Put
    at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:96)
    at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:55)
    at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:154)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174)
    at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:563)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
    at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:215)
*My Mapper class:*

public static class Mapper1 extends TableMapper<Text, IntArrayWritable>
{
     public void map(ImmutableBytesWritable row, Result values, Context
context) throws IOException
     {
       ......
       ......
       context.write(cookie, out); //out is an IntArrayWritable
     }
}

---------------------------------------------------------------------------------------------------------------------------------------
*
Reducer Class:*

public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
Text>
{
        public void reduce(Text key, Iterator <IntArrayWritable> values,
Context context)
                throws IOException, InterruptedException
        {
           ....
           ....
            Put put = new Put(rowid.getBytes());
            put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
Bytes.toBytes(val));
            context.write(new Text(rowid), put);
        }
}

-----------------------------------------------------------------------------------------------------------------------------------------

*Main()*

public static void main(String[] args) throws Exception
{
        HBaseConfiguration conf = new HBaseConfiguration();
        Job job = new Job(conf, "CookieJob");
        job.setJarByClass(CookieJob.class);
        scan.addFamily("cf".getBytes()); // The InputTable has only 1 column
family - cf.
        scan.setMaxVersions();
        TableMapReduceUtil.initTableMapperJob("InputTable", scan,
Mapper1.class, Text.class,
        IntArrayWritable.class, job);
        TableMapReduceUtil.initTableReducerJob("OutputTable",
Reducer1.class, job);
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

---------------------------------------------------------------------------------------------------------------------------------------

Please let me know if somebody knows what could be the reason behind this. I
have been bogged down by this exception for a very long time.

If somebody could throw light on why this exception is occurring before the
reduce starts, I would be very much thankful.
Please let me know if you need any other infomation to debug this issue.

Regards,
Narayanan

On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <[EMAIL PROTECTED]> wrote:
> Hi Everyone,
>
> I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink both
> being HBase Tables.
>

Please upgrade.  Its hard to help you when you run a version so old.
None of us remember how it works.  At least retry with 0.20.6.  Better
still, move to 0.90.x.
St.Ack

I found this older thread that _might_ help you ... but as Stack says,
better to upgrade to 0.90.x if possible.
http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch

--Suraj

On Tue, Jul 26, 2011 at 11:25 AM, Stack <[EMAIL PROTECTED]> wrote:
> On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <[EMAIL PROTECTED]> wrote:
>> Hi Everyone,
>>
>> I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink both
>> being HBase Tables.
>>
>
> Please upgrade.  Its hard to help you when you run a version so old.
> None of us remember how it works.  At least retry with 0.20.6.  Better
> still, move to 0.90.x.
> St.Ack
>

Hi Stack/Suraj,

I tried my MR code on HBase to 0.90.x version.

But I am getting the same exception after the Map stage is complete:
*java.io.IOException:
Pass a Delete or a Put*.

I did a search on web and found the *TableOutputFormat* source code where
the *write* method is throwing the "*Pass a Delete or Put*" error:
http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
(Line 92-96)

>From this, I understand that the value in Reduce Output key,value pair
should be an instance of Put or Delete.

My Reduce is also throwing an instance of Put :

public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
Text>
{
        public void reduce(Text key, Iterator <IntArrayWritable> values,
Context context)
                throws IOException, InterruptedException
        {
           ....
           ....
            Put put = new Put(rowid.getBytes());
            put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
Bytes.toBytes(val));
            context.write(new Text(rowid), put);
        }
}

Then why am I getting this exception *java.io.IOException: Pass a Delete or
a Put*?

Any insights into what I am missing here would be really helpful.

Thanks,
Narayanan

On Wed, Jul 27, 2011 at 12:07 AM, Suraj Varma <[EMAIL PROTECTED]> wrote:

> I found this older thread that _might_ help you ... but as Stack says,
> better to upgrade to 0.90.x if possible.
>
> http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch
>
> --Suraj
>
> On Tue, Jul 26, 2011 at 11:25 AM, Stack <[EMAIL PROTECTED]> wrote:
> > On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <[EMAIL PROTECTED]>
> wrote:
> >> Hi Everyone,
> >>
> >> I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink
> both
> >> being HBase Tables.
> >>
> >
> > Please upgrade.  Its hard to help you when you run a version so old.
> > None of us remember how it works.  At least retry with 0.20.6.  Better
> > still, move to 0.90.x.
> > St.Ack
> >
>

Study the mapreduce examples in unit tests or under our mapreduce
package.  Below looks fine to me.  Maybe its how the job is
configured.

St.Ack
P.S. you don't have to find our src in random locations; e.g. our paul
smiths' apache home dir.  Our src is here: hbase.org

St.Ack

On Fri, Jul 29, 2011 at 8:53 AM, Narayanan K <[EMAIL PROTECTED]> wrote:
> Hi Stack/Suraj,
>
> I tried my MR code on HBase to 0.90.x version.
>
> But I am getting the same exception after the Map stage is complete:
> *java.io.IOException:
> Pass a Delete or a Put*.
>
> I did a search on web and found the *TableOutputFormat* source code where
> the *write* method is throwing the "*Pass a Delete or Put*" error:
> http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
> (Line 92-96)
>
> From this, I understand that the value in Reduce Output key,value pair
> should be an instance of Put or Delete.
>
> My Reduce is also throwing an instance of Put :
>
> public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
> Text>
> {
>        public void reduce(Text key, Iterator <IntArrayWritable> values,
> Context context)
>                throws IOException, InterruptedException
>        {
>           ....
>           ....
>            Put put = new Put(rowid.getBytes());
>            put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
>            context.write(new Text(rowid), put);
>        }
> }
>
> Then why am I getting this exception *java.io.IOException: Pass a Delete or
> a Put*?
>
> Any insights into what I am missing here would be really helpful.
>
> Thanks,
> Narayanan
>
>
>
>
>
> On Wed, Jul 27, 2011 at 12:07 AM, Suraj Varma <[EMAIL PROTECTED]> wrote:
>
>> I found this older thread that _might_ help you ... but as Stack says,
>> better to upgrade to 0.90.x if possible.
>>
>> http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch
>>
>> --Suraj
>>
>> On Tue, Jul 26, 2011 at 11:25 AM, Stack <[EMAIL PROTECTED]> wrote:
>> > On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <[EMAIL PROTECTED]>
>> wrote:
>> >> Hi Everyone,
>> >>
>> >> I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink
>> both
>> >> being HBase Tables.
>> >>
>> >
>> > Please upgrade.  Its hard to help you when you run a version so old.
>> > None of us remember how it works.  At least retry with 0.20.6.  Better
>> > still, move to 0.90.x.
>> > St.Ack
>> >
>>
>

Hi

Many Thanks to all who responded!!

I sat the whole day to solve this and to my delight, I finally found the
issue & pinpointed where this exception kept on recurring.

Though my explanation may not make sense, but this is what I changed in my
existing code to get the reducer running fine:
(I just changed *Iterator* to *Iterable*.)
*Erronious reduce format
*
public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
IntArrayWritable, ImmutableBytesWritable> {

        public void reduce(ImmutableBytesWritable key, Iterator
<IntArrayWritable> values, Context context)
                throws IOException, InterruptedException {

        while (values.hasNext())
        {
          ....
        }
            Put put = new Put(key.get());
            put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
Bytes.toBytes(val));
            context.write(key, put);
        }
}

*Corrected/Working fine reduce format*

public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
IntArrayWritable, ImmutableBytesWritable> {

        public void reduce(ImmutableBytesWritable key,
*Iterable*<IntArrayWritable>values,
Context context)
                throws IOException, InterruptedException {

                 * for (IntArrayWritable temp : values)
                  {*
                     ....
                  }
                 Put put = new Put(key.get());
                 put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
Bytes.toBytes(val));
                 context.write(key, put);
        }
}

--------------------------------------------------------------------------------------------------------------------------------------------------------------

Please let me know if this could be the real reason? If so, why it doesnt
support Iterator for values in reduce?

Thanks,
Narayanan

On Fri, Jul 29, 2011 at 10:02 PM, Stack <[EMAIL PROTECTED]> wrote:

> Study the mapreduce examples in unit tests or under our mapreduce
> package.  Below looks fine to me.  Maybe its how the job is
> configured.
>
> St.Ack
> P.S. you don't have to find our src in random locations; e.g. our paul
> smiths' apache home dir.  Our src is here: hbase.org
>
> St.Ack
>
> On Fri, Jul 29, 2011 at 8:53 AM, Narayanan K <[EMAIL PROTECTED]>
> wrote:
> > Hi Stack/Suraj,
> >
> > I tried my MR code on HBase to 0.90.x version.
> >
> > But I am getting the same exception after the Map stage is complete:
> > *java.io.IOException:
> > Pass a Delete or a Put*.
> >
> > I did a search on web and found the *TableOutputFormat* source code where
> > the *write* method is throwing the "*Pass a Delete or Put*" error:
> >
> http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
> > (Line 92-96)
> >
> > From this, I understand that the value in Reduce Output key,value pair
> > should be an instance of Put or Delete.
> >
> > My Reduce is also throwing an instance of Put :
> >
> > public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
> > Text>
> > {
> >        public void reduce(Text key, Iterator <IntArrayWritable> values,
> > Context context)
> >                throws IOException, InterruptedException
> >        {
> >           ....
> >           ....
> >            Put put = new Put(rowid.getBytes());
> >            put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> > Bytes.toBytes(val));
> >            context.write(new Text(rowid), put);
> >        }
> > }
> >
> > Then why am I getting this exception *java.io.IOException: Pass a Delete
> or
> > a Put*?
> >
> > Any insights into what I am missing here would be really helpful.
> >
> > Thanks,
> > Narayanan
> >
> >
> >
> >
> >
> > On Wed, Jul 27, 2011 at 12:07 AM, Suraj Varma <[EMAIL PROTECTED]>
> wrote:
> >
> >> I found this older thread that _might_ help you ... but as Stack says,
> >> better to upgrade to 0.90.x if possible.
> >>
> >>
> http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch

Yes - this was also the reason in the thread I had linked earlier.
The reason is that there are two versions of the Reducer. One in the
mapred package (that uses Iterator) and the new one in mapreduce
package (that uses Iterable). It is likely that you used the new one
with the old reduce signature.

http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapred/Reducer.html#reduce(K2,
java.util.Iterator, org.apache.hadoop.mapred.OutputCollector,
org.apache.hadoop.mapred.Reporter)
http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapreduce/Reducer.html#reduce(KEYIN,
java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer.Context)

Hope that clarifies.
--Suraj

On Sat, Jul 30, 2011 at 7:58 AM, Narayanan K <[EMAIL PROTECTED]> wrote:
> Hi
>
> Many Thanks to all who responded!!
>
> I sat the whole day to solve this and to my delight, I finally found the
> issue & pinpointed where this exception kept on recurring.
>
> Though my explanation may not make sense, but this is what I changed in my
> existing code to get the reducer running fine:
> (I just changed *Iterator* to *Iterable*.)
>
>
> *Erronious reduce format
> *
> public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
> IntArrayWritable, ImmutableBytesWritable> {
>
>        public void reduce(ImmutableBytesWritable key, Iterator
> <IntArrayWritable> values, Context context)
>                throws IOException, InterruptedException {
>
>        while (values.hasNext())
>        {
>          ....
>        }
>            Put put = new Put(key.get());
>            put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
>            context.write(key, put);
>        }
> }
>
> *Corrected/Working fine reduce format*
>
> public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
> IntArrayWritable, ImmutableBytesWritable> {
>
>        public void reduce(ImmutableBytesWritable key,
> *Iterable*<IntArrayWritable>values,
> Context context)
>                throws IOException, InterruptedException {
>
>                 * for (IntArrayWritable temp : values)
>                  {*
>                     ....
>                  }
>
>
>                 Put put = new Put(key.get());
>                 put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
>                 context.write(key, put);
>        }
> }
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Please let me know if this could be the real reason? If so, why it doesnt
> support Iterator for values in reduce?
>
> Thanks,
> Narayanan
>
> On Fri, Jul 29, 2011 at 10:02 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>> Study the mapreduce examples in unit tests or under our mapreduce
>> package.  Below looks fine to me.  Maybe its how the job is
>> configured.
>>
>> St.Ack
>> P.S. you don't have to find our src in random locations; e.g. our paul
>> smiths' apache home dir.  Our src is here: hbase.org
>>
>> St.Ack
>>
>> On Fri, Jul 29, 2011 at 8:53 AM, Narayanan K <[EMAIL PROTECTED]>
>> wrote:
>> > Hi Stack/Suraj,
>> >
>> > I tried my MR code on HBase to 0.90.x version.
>> >
>> > But I am getting the same exception after the Map stage is complete:
>> > *java.io.IOException:
>> > Pass a Delete or a Put*.
>> >
>> > I did a search on web and found the *TableOutputFormat* source code where
>> > the *write* method is throwing the "*Pass a Delete or Put*" error:
>> >
>> http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
>> > (Line 92-96)
>> >
>> > From this, I understand that the value in Reduce Output key,value pair
>> > should be an instance of Put or Delete.
>> >
>> > My Reduce is also throwing an instance of Put :
>> >
>> > public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
>> > Text>
>> > {