Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Avro + Snappy changing blocksize of snappy compression


Copy link to this message
-
Re: Avro + Snappy changing blocksize of snappy compression
Try a range from smaller block sizes (4k) and up.  256K is a larger block
size than many compression codecs are sensitive to.

Also for reference, try it with the deflate codec at a couple different
compression levels -- 1, 3, 5, and 7 should show a trend with respect to
block size.  As the compression level increases, the compressor can take
advantage of larger blocks.

In the deflate/gzip case that I have explored heavily, the effectiveness
of the block size also varies significantly depending on the
characteristics of the data being compressed.
(note: gzip uses deflate compression)

On 4/18/12 1:33 PM, "snikhil0" <[EMAIL PROTECTED]> wrote:

>I had tried the sync Interval as well and I get the same results: meaning
>no
>change in final avro data file.
>
>Nikhil
>
>--
>View this message in context:
>http://apache-avro.679487.n3.nabble.com/Avro-Snappy-changing-blocksize-of-
>snappy-compression-tp3920732p3921256.html
>Sent from the Avro - Users mailing list archive at Nabble.com.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB