Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Avro + Snappy changing blocksize of snappy compression


+
snikhil0 2012-04-18, 17:23
+
Tatu Saloranta 2012-04-18, 17:52
Copy link to this message
-
Re: Avro + Snappy changing blocksize of snappy compression
Hey Nikhil,

When using Avro Datafiles, you perhaps need to tweak its sync-interval
to affect compression chunk sizes:
http://avro.apache.org/docs/1.6.3/api/java/org/apache/avro/file/DataFileWriter.html#setSyncInterval(int)

On Wed, Apr 18, 2012 at 10:53 PM, snikhil0 <[EMAIL PROTECTED]> wrote:
> I am experimenting with Avro and snappy and want to plot the size of the
> compressed avro datafile as a function of varying compression block size. I
> am doing this by setting the configuration value for
> "io.compression.codec.snappy.buffersize". Unfortunately, this is not
> working: or more precisely for buffer sizes 256K to 2MB I get the same size
> output avro (snappyfied) data file. What am I missing? Someone had success
> with this?
>
> Thanks,
> Nikhil
>
> --
> View this message in context: http://apache-avro.679487.n3.nabble.com/Avro-Snappy-changing-blocksize-of-snappy-compression-tp3920732p3920732.html
> Sent from the Avro - Users mailing list archive at Nabble.com.

--
Harsh J
+
snikhil0 2012-04-18, 20:33
+
Scott Carey 2012-04-18, 21:18
+
Tatu Saloranta 2012-04-18, 21:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB