I'm using HBase with MapReduce to load a lot of data, so I have decide to do it with bulk load. I parse my keys with SHA1, but when I try to load them, I got this exception.
java.io.IOException: Added a key not lexically larger than previous key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3>\xC9\xC7\x0E, lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207) at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324) at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289) at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206) at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168) at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
I work with HBase 0.94.6. I have been loking for if I could define any reducer, since, I have defined no one. I have read something about KeyValueSortReducer but, I don'tknow if there's something that extends TableReducer or I'm lookging for a wrong way.
Yes, I do, job.setMapperClass(EventMapper.class); job.setMapOutputKeyClass(ImmutableBytesWritable.class); job.setMapOutputValueClass(KeyValue.class);
FileOutputFormat.setOutputPath(job, hbasePath); HTable table = new HTable(jConf, MEM_TABLE_HBASE); HFileOutputFormat.configureIncrementalLoad(job, table); The error is happeing in a MRUnit, I don't know if it changes something about the behavior, because I had some troubles in the past for the same reason about the serialization in Hbase 0.96 and MRUnit. . Besides, in the setup of the MRUnit test I load some data in hbase with keys in sha1 and it works.
El jueves, 1 de mayo de 2014, Jean-Daniel Cryans <[EMAIL PROTECTED]> escribió:
The error was that when I was emitting the <K,V>, I was doing SHA about K, not about the key in the Value. The Value is a KeyValue and here's where I had to do the SHA1. 2014-05-02 0:42 GMT+02:00 Guillermo Ortiz <[EMAIL PROTECTED]>:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext