Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> how is the partitioner working when compression is enabled?


Copy link to this message
-
how is the partitioner working when compression is enabled?
I am using Kafka 0.8 and message compression can only be enabled on producer side.

However one batch of messages (decided by the producer configuration batch.num.messages) is compressed as a single message and stored at the broker side. I think one batch compressed message will be stored at one broker.

I am wondering in this case how the producer side partitioner class will work? The batch will be partitioned to the partition of the first or last message in the batch?
Lex
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB