Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Generating snappy compressed avro files as hadoop map reduce input files


Copy link to this message
-
Generating snappy compressed avro files as hadoop map reduce input files
David Ginzburg 2013-10-13, 15:16
Hi,

I am writing an application that produces avro record files , to be stored on AWS S3 as possible input to EMR.
I would like to compress with snappy codec before storing them on S3.
It is my understanding that hadoop currently uses a different snappy codec, mostly used as intermediate map output format .
My question is how can I generate within my application logic (not MR) snappy compressed avro files?

+
Bertrand Dechoux 2013-10-13, 15:36
+
David Ginzburg 2013-10-13, 15:58
+
graham sanderson 2013-10-13, 18:16
+
David Ginzburg 2013-10-13, 19:23
+
graham sanderson 2013-10-13, 21:31
+
Bertrand Dechoux 2013-10-14, 07:49