Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Review Request 24214: Patch for KAFKA-1374


Copy link to this message
-
Re: Review Request 24214: Patch for KAFKA-1374

This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24214/#review50128

core/src/main/scala/kafka/log/LogCleaner.scala
<https://reviews.apache.org/r/24214/#comment87704>

    Thinking about this a bit more. I am wondering if it would be better if we introduce a per-topic level log.compact.compress.codec property. During log compaction, we always write the retained data using the specified compress codec, independent of whether the original records are compressed or not. This provides the following benefits.
    
    1. Whether the messages were compressed originally, they can be compressed on the broker side over time. Since compact topics preserve records much longer, enabling compression on the broker side will be beneficial in general.
    
    2. As old records are removed, we still want to batch enough messages to do the compression.
    
    3. The code can be a bit simpler. We can just (deep) iterate messages (using MemoryRecods.iterator) and append retained messages to an output MemoryRecords. The output MemoryRecords will be initialized with the configured compress codec and batch size.
- Jun Rao
On Aug. 9, 2014, 10:51 a.m., Manikumar Reddy O wrote: