Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - map/reduce of compressed Avro


Copy link to this message
-
Re: map/reduce of compressed Avro
Martin Kleppmann 2013-04-22, 14:52
You don't need to do anything special to accept compressed Avro files as
input, as it's detected automatically and decompressed transparently. M/R
jobs support all codecs that the Java implementation supports; at the
moment I think that's deflate, snappy and bzip2.

If you want to generate compressed output, use
FileOutputFormat.setCompressOutput(job, true);

Martin
On 22 April 2013 02:26, nir_zamir <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Does anyone know if/how it's possible to get compressed Avro files as an
> input to a M/R job?
> If so, which codecs are supported?
>
> Thanks,
> Nir
>
>
>
> --
> View this message in context:
> http://apache-avro.679487.n3.nabble.com/map-reduce-of-compressed-Avro-tp4026947.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>