Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> map/reduce of compressed Avro


Copy link to this message
-
Re: map/reduce of compressed Avro
You don't need to do anything special to accept compressed Avro files as
input, as it's detected automatically and decompressed transparently. M/R
jobs support all codecs that the Java implementation supports; at the
moment I think that's deflate, snappy and bzip2.

If you want to generate compressed output, use
FileOutputFormat.setCompressOutput(job, true);

Martin
On 22 April 2013 02:26, nir_zamir <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Does anyone know if/how it's possible to get compressed Avro files as an
> input to a M/R job?
> If so, which codecs are supported?
>
> Thanks,
> Nir
>
>
>
> --
> View this message in context:
> http://apache-avro.679487.n3.nabble.com/map-reduce-of-compressed-Avro-tp4026947.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB