Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase MR - key/value mismatch


+
Omkar Joshi 2013-09-05, 06:16
+
Shahab Yunus 2013-09-05, 13:25
+
Ted Yu 2013-09-05, 14:34
+
Shahab Yunus 2013-09-05, 16:46
+
Ted Yu 2013-09-05, 16:57
Copy link to this message
-
Re: HBase MR - key/value mismatch
Thanks Ted.

Regards,
Shahab
On Thu, Sep 5, 2013 at 12:57 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> The reducer also serves as combiner whose output would be sent to reducer.
>
>             org.apache.hadoop.mapreduce.Reducer<Text, Text,
> ImmutableBytesWritable, org.apache.hadoop.io.Writable>.Context context)
>
> So the type parameters above should facilitate this.
> Take a look at the PutCombiner from HBase source code:
>
> public class PutCombiner<K> extends Reducer<K, Put, K, Put> {
> Cheers
>
> On Thu, Sep 5, 2013 at 9:46 AM, Shahab Yunus <[EMAIL PROTECTED]>
> wrote:
>
> > Ted,
> >
> > Might be a something very basic that I am missing but why should OP's
> > reducer's key be of type ImmutableBytesWritable if he is emitting Text in
> > the mapper? Thanks.
> >
> >  protected void map(
> >
> >             ImmutableBytesWritable key,
> >
> >             Result value,
> >
> >             org.apache.hadoop.mapreduce.Mapper<ImmutableBytesWritable,
> > Result, Text, Text>.Context context)
> >
> >             throws IOException, InterruptedException {
> >
> > context.write(this.sentenseOriginal, this.sentenseParsed);
> > //sentenseOriginal
> > is Text
> >
> >
> > Regards,
> > Shahab
> >
> >
> > On Thu, Sep 5, 2013 at 10:34 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > public class SentimentCalculationHBaseReducer extends
> > >
> > >         TableReducer<Text, Text, ImmutableBytesWritable> {
> > >
> > > The first type parameter for reducer should be ImmutableBytesWritable
> > >
> > > Cheers
> > >
> > >
> > > On Wed, Sep 4, 2013 at 11:16 PM, Omkar Joshi <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > I'm trying to execute a MR code over stand-alone HBase(0.94.11). I
> had
> > > > read the HBase api and modified my MR code to read data and getting
> > > > exceptions in the Reduce phase.
> > > >
> > > > The exception I get is :
> > > >
> > > > 13/09/05 16:16:17 INFO mapred.JobClient:  map 0% reduce 0%
> > > >
> > > > 13/09/05 16:23:31 INFO mapred.JobClient: Task Id :
> > > > attempt_201309051437_0005_m_000000_0, Status : FAILED
> > > >
> > > > java.io.IOException: wrong key class: class
> > > > org.apache.hadoop.hbase.io.ImmutableBytesWritable is not class
> > > > org.apache.hadoop.io.Text
> > > >
> > > >         at
> org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:164)
> > > >
> > > >         at
> > > >
> > >
> >
> org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1168)
> > > >
> > > >         at
> > > >
> > >
> >
> org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1492)
> > > >
> > > >         at
> > > >
> > >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > > >
> > > >         at
> > > >
> > >
> >
> com.hbase.mapreduce.SentimentCalculationHBaseReducer.reduce(SentimentCalculationHBaseReducer.java:199)
> > > >
> > > >         at
> > > >
> > >
> >
> com.hbase.mapreduce.SentimentCalculationHBaseReducer.reduce(SentimentCalculationHBaseReducer.java:1)
> > > >
> > > >         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> > > >
> > > >         at
> > > >
> org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1513)
> > > >
> > > >         at
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1436)
> > > >
> > > >         at
> > > >
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
> > > >
> > > >         at
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
> > > >
> > > >         at
> > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
> > > >
> > > >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > > >
> > > >         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > > >
> > > >         at java.security.AccessController.doPrivileged(Native Method)
> > > >
> > > >         at javax.security.auth.Subject.doAs(Subject.java:415)
+
Ted Yu 2013-09-06, 20:01