Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Review Request 17263: New producer for Kafka.

Copy link to this message
Re: Review Request 17263: New producer for Kafka.

> On Jan. 26, 2014, 11:30 p.m., Sriram Subramanian wrote:
> > clients/src/main/java/kafka/clients/producer/internals/BufferPool.java, line 80
> > <https://reviews.apache.org/r/17263/diff/1/?file=436452#file436452line80>
> >
> >     What happens if freeUp actually got memory from the free list? We still seem to create a new buffer with the given size. Why are we not returning from the pool? It looks like availableMemory can go negative here.
> Jay Kreps wrote:
>     Let me know if this is just confusion about how the pool works. The idea is that I want to optimize the common case of allocations that match the batch size, but I still want to allow large allocations of any size. I also want to avoid implementing malloc, as you rightly pointed out. The compromise I came up with was to special case the batch size and pool that size since that should be 99% of the allocations, when messages come in that are bigger then this i just directly allocate memory from them and don't attempt to pool it.
>     But this means you can have a bunch of buffers in the free list (which are always exactly batchSize) that use up all available memory. So if you get a custom allocation size you need to toss out some pooled buffers to ensure you don't exceed your memory limit. Let me know if that makes sense and isn't totally insane.

I would be surprised if one sets the batch size and sends messages larger than the batch size. As you pointed out this would be rare. Does it make sense then to keep the max memory a soft limit and just allocate the large message and reduce complexity here. I am not recommending one way or the other though.
- Sriram
This is an automatically generated e-mail. To reply, visit:
On Jan. 23, 2014, 8:54 p.m., Jay Kreps wrote:
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17263/
> -----------------------------------------------------------
> (Updated Jan. 23, 2014, 8:54 p.m.)
> Review request for kafka.
> Bugs: KAFKA-1227
>     https://issues.apache.org/jira/browse/KAFKA-1227
> Repository: kafka
> Description
> -------
> KAFKA-1227 New producer!
> Diffs
> -----
>   clients/build.sbt PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/BufferExhaustedException.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/Callback.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/DefaultPartitioner.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/KafkaProducer.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/MockProducer.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/Partitioner.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/Producer.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/ProducerConfig.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/ProducerRecord.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/RecordSend.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/BufferPool.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/Metadata.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/ProduceRequestResult.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/RecordAccumulator.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/RecordBatch.java PRE-CREATION
>   clients/src/main/java/kafka/clients/producer/internals/Sender.java PRE-CREATION
>   clients/src/main/java/kafka/clients/tools/ProducerPerformance.java PRE-CREATION
>   clients/src/main/java/kafka/common/ByteSerialization.java PRE-CREATION
>   clients/src/main/java/kafka/common/Cluster.java PRE-CREATION