Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request: SQOOP-428: Support compression for Avro import

Copy link to this message
Re: Review Request: SQOOP-428: Support compression for Avro import

This is an automatically generated e-mail. To reply, visit:

(Updated 2012-01-31 09:50:52.475912)
Review request for Sqoop.

Adds the option of providing a Codec class name as well.

This basically only ports all the code from Avro's (1.5.4) AvroOutputFormat to the new MR API.

I've changed the test to extract the common functionality into a helper method because they are the same apart from the two command line arguments.

I could have deleted AvroJob completely but as I was told last time that binary compatibility needs to be maintained I left it in. It's not needed anymore as all necessary functionality can be gotten from Avro's own version of that file as far as I can tell. So if it's okay to delete that redundant file (two actually, cloudera and apache package) let me know and I'll provide a new patch.
This addresses bug SQOOP-428.
Diffs (updated)

  src/java/com/cloudera/sqoop/io/CodecMap.java ffe949b
  src/java/org/apache/sqoop/io/CodecMap.java 5b67206
  src/java/org/apache/sqoop/mapreduce/AvroJob.java a57aaf1
  src/java/org/apache/sqoop/mapreduce/AvroOutputFormat.java 96befd7
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java ed6954a
  src/test/com/cloudera/sqoop/TestAvroImport.java 1b8b046
  src/test/com/cloudera/sqoop/io/TestCodecMap.java f2f4039

Diff: https://reviews.apache.org/r/3600/diff

All tests pass for hadoopversion=20 but TestColumnTypes fails for me on 23. I can't see how that's related though.