Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase MR - key/value mismatch


Copy link to this message
-
Re: HBase MR - key/value mismatch
The reducer also serves as combiner whose output would be sent to reducer.

            org.apache.hadoop.mapreduce.Reducer<Text, Text,
ImmutableBytesWritable, org.apache.hadoop.io.Writable>.Context context)

So the type parameters above should facilitate this.
Take a look at the PutCombiner from HBase source code:

public class PutCombiner<K> extends Reducer<K, Put, K, Put> {
Cheers

On Thu, Sep 5, 2013 at 9:46 AM, Shahab Yunus <[EMAIL PROTECTED]> wrote:

> Ted,
>
> Might be a something very basic that I am missing but why should OP's
> reducer's key be of type ImmutableBytesWritable if he is emitting Text in
> the mapper? Thanks.
>
>  protected void map(
>
>             ImmutableBytesWritable key,
>
>             Result value,
>
>             org.apache.hadoop.mapreduce.Mapper<ImmutableBytesWritable,
> Result, Text, Text>.Context context)
>
>             throws IOException, InterruptedException {
>
> context.write(this.sentenseOriginal, this.sentenseParsed);
> //sentenseOriginal
> is Text
>
>
> Regards,
> Shahab
>
>
> On Thu, Sep 5, 2013 at 10:34 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > public class SentimentCalculationHBaseReducer extends
> >
> >         TableReducer<Text, Text, ImmutableBytesWritable> {
> >
> > The first type parameter for reducer should be ImmutableBytesWritable
> >
> > Cheers
> >
> >
> > On Wed, Sep 4, 2013 at 11:16 PM, Omkar Joshi <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > I'm trying to execute a MR code over stand-alone HBase(0.94.11). I had
> > > read the HBase api and modified my MR code to read data and getting
> > > exceptions in the Reduce phase.
> > >
> > > The exception I get is :
> > >
> > > 13/09/05 16:16:17 INFO mapred.JobClient:  map 0% reduce 0%
> > >
> > > 13/09/05 16:23:31 INFO mapred.JobClient: Task Id :
> > > attempt_201309051437_0005_m_000000_0, Status : FAILED
> > >
> > > java.io.IOException: wrong key class: class
> > > org.apache.hadoop.hbase.io.ImmutableBytesWritable is not class
> > > org.apache.hadoop.io.Text
> > >
> > >         at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:164)
> > >
> > >         at
> > >
> >
> org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1168)
> > >
> > >         at
> > >
> >
> org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1492)
> > >
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > >
> > >         at
> > >
> >
> com.hbase.mapreduce.SentimentCalculationHBaseReducer.reduce(SentimentCalculationHBaseReducer.java:199)
> > >
> > >         at
> > >
> >
> com.hbase.mapreduce.SentimentCalculationHBaseReducer.reduce(SentimentCalculationHBaseReducer.java:1)
> > >
> > >         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> > >
> > >         at
> > > org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1513)
> > >
> > >         at
> > >
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1436)
> > >
> > >         at
> > >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
> > >
> > >         at
> > >
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
> > >
> > >         at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
> > >
> > >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > >
> > >         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > >
> > >         at java.security.AccessController.doPrivileged(Native Method)
> > >
> > >         at javax.security.auth.Subject.doAs(Subject.java:415)
> > >
> > >         at
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> > >
> > >         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > >
> > >
> > >
> > > Providing the partial(excluding the business logic) codes
> > >
> > >
> > > Mapper:
> > >
> > >
> > > public class SentimentCalculationHBaseMapper extends TableMapper<Text,