Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Uncompress / re-compress of messages in the message server


Copy link to this message
-
Re: Uncompress / re-compress of messages in the message server
Yes, your understanding is correct. The reason we have to recompress the
messages is to assign a unique offset to messages inside a compressed
message. Some preliminary load testing shows 30% increase in CPU, but that
is using GZIP which is known to be CPU intensive. By this week, we will
know the CPU usage for a lighter compression codec like Snappy. Will post
the results on the mailing list.

Thanks,
Neha

On Monday, March 18, 2013, Ross Black wrote:

> Hi,
>
> I have just started looking at moving from 0.7 to 0.8 and wanted to confirm
> my understanding of code in the message server/broker.
>
> In the code for 0.8, KafkaApis.appendToLocalLog calls log.append(...,
> assignOffsets = true), which then calls ByteBufferMessageSet.assignOffsets.
> This method seems to uncompress and then re-compress the entire set of
> messages.
>
> Is my understanding of the code correct?
> Has any testing been done on the CPU consumption / performance of the
> message server to determine whether this adversely impacts message
> throughput under high load?
>
> Thanks,
> Ross
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB