Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Avro + Snappy changing blocksize of snappy compression


Copy link to this message
-
Re: Avro + Snappy changing blocksize of snappy compression
Scott Carey 2012-04-18, 21:18
Try a range from smaller block sizes (4k) and up.  256K is a larger block
size than many compression codecs are sensitive to.

Also for reference, try it with the deflate codec at a couple different
compression levels -- 1, 3, 5, and 7 should show a trend with respect to
block size.  As the compression level increases, the compressor can take
advantage of larger blocks.

In the deflate/gzip case that I have explored heavily, the effectiveness
of the block size also varies significantly depending on the
characteristics of the data being compressed.
(note: gzip uses deflate compression)

On 4/18/12 1:33 PM, "snikhil0" <[EMAIL PROTECTED]> wrote:

>I had tried the sync Interval as well and I get the same results: meaning
>no
>change in final avro data file.
>
>Nikhil
>
>--
>View this message in context:
>http://apache-avro.679487.n3.nabble.com/Avro-Snappy-changing-blocksize-of-
>snappy-compression-tp3920732p3921256.html
>Sent from the Avro - Users mailing list archive at Nabble.com.