Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Low latency topics


Copy link to this message
-
Re: Low latency topics
Neha Narkhede 2012-11-08, 02:58
Bob,

The latency that you are seeing seems a little high. I would suspect
at your low write rate, the # of partitions is too high.
The simplest test for latency could be one producer, one broker
partition and one consumer. It will be great if you can give that a
try.

Thanks,
Neha

On Wed, Nov 7, 2012 at 5:55 PM, Bob Cotton <[EMAIL PROTECTED]> wrote:
> Hello,
>
> We have a low-volume topic (~75msgs/sec) for which we would like to have a
> low propagation delay from producer to consumer.
>
> We have 3 brokers, each with a default of 4 partitions each. for a total of
> 12 partitions.
> The producer is sync, without compression. There are 8 producers each
> producing 1/8 of the traffic.
> We are using the high-level java consumer, with 4 threads consuming the
> topic.
>
> We are wrapping the message with a custom Encoder/Decoder and record
> currentTimeMillis() on the sender, and do the same in the receiver, then
> record the propagation delay. All hosts are time synced with ntp.
>
> With the settings on the broker for flush messages and flush interval
> (unset, defaults to 500 msgs and 3000ms) the overall 95th percentile for
> propagation is 2,500ms.
>
> When we adjust the topic flush interval to 20ms, the 95th percentile drops
> to 1,700ms
> When we adjust the consumers "fetcher.backoff.ms" to 10, the 95th
> percentile drops to about 970ms.
>
> We would like this to be sub-500ms.
> We could run with less partitions and/or more consumer threads.
>
> Anything glaring about this config? anything we're missing?
>
> Thanks
> -Bob