Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Review Request 17263: New producer for Kafka.


Copy link to this message
-
Re: Review Request 17263: New producer for Kafka.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17263/#review32863
-----------------------------------------------------------

clients/src/main/java/kafka/clients/producer/internals/BufferPool.java
<https://reviews.apache.org/r/17263/#comment61868>

    General comment - We are using the batch size to be the pool size. I had done a very similar implementation previously and had found it led to a lot of fragmentation. The main problem is that you dont want to let the pool size be determined by a batch size that can change. If the batch size is increased, a low volume topic partition is going to use up batch size of memory and other requests would stall. I am not proposing we implement malloc but I think we can do better than this. The poolable size should be a fixed value and much smaller than the batch size. The batch size can then be served as a multiple of the poolable size. This ensures that increasing batch size does not affect your allocation pattern and helps you to keep the poolable size much smaller which is important to support the low volume topic. At the least, we should track effective memory used Vs total available memory using metrics.
- Sriram Subramanian
On Jan. 23, 2014, 8:54 p.m., Jay Kreps wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17263/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 8:54 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1227
>     https://issues.apache.org/jira/browse/KAFKA-1227
>
>
> Repository: kafka
>
>
> Description
> -------
>
> KAFKA-1227 New producer!
>
>
> Diffs
> -----
>
>   clients/build.sbt PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/BufferExhaustedException.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/Callback.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/DefaultPartitioner.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/KafkaProducer.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/MockProducer.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/Partitioner.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/Producer.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/ProducerConfig.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/ProducerRecord.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/RecordSend.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/BufferPool.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/Metadata.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/ProduceRequestResult.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/RecordAccumulator.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/RecordBatch.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/Sender.java PRE-CREATION
>   clients/src/main/java/kafka/clients/tools/ProducerPerformance.java PRE-CREATION
>   clients/src/main/java/kafka/common/ByteSerialization.java PRE-CREATION
>   clients/src/main/java/kafka/common/Cluster.java PRE-CREATION
>   clients/src/main/java/kafka/common/Configurable.java PRE-CREATION
>   clients/src/main/java/kafka/common/Deserializer.java PRE-CREATION
>   clients/src/main/java/kafka/common/KafkaException.java PRE-CREATION
>   clients/src/main/java/kafka/common/Metric.java PRE-CREATION
>   clients/src/main/java/kafka/common/Node.java PRE-CREATION
>   clients/src/main/java/kafka/common/PartitionInfo.java PRE-CREATION
>   clients/src/main/java/kafka/common/Serializer.java PRE-CREATION
>   clients/src/main/java/kafka/common/StringSerialization.java PRE-CREATION