Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Generating snappy compressed avro files as hadoop map reduce input files


Copy link to this message
-
Generating snappy compressed avro files as hadoop map reduce input files
Hi,

I am writing an application that produces avro record files , to be stored on AWS S3 as possible input to EMR.
I would like to compress with snappy codec before storing them on S3.
It is my understanding that hadoop currently uses a different snappy codec, mostly used as intermediate map output format .
My question is how can I generate within my application logic (not MR) snappy compressed avro files?

+
Bertrand Dechoux 2013-10-13, 15:36
+
David Ginzburg 2013-10-13, 15:58
+
graham sanderson 2013-10-13, 18:16
+
David Ginzburg 2013-10-13, 19:23
+
graham sanderson 2013-10-13, 21:31
+
Bertrand Dechoux 2013-10-14, 07:49
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB