Hi, Colin,

When fetching data for a partition, the leader needs to translate the fetch
offset to a position in a log segment with an index lookup. If the fetch
request now also needs to cache the offset for the next fetch request,
there will be an extra offset index lookup. The offset index lookup can
potentially be expensive since it could require disk I/Os. One way to
optimize this a bit is to further cache the log segment position for the
next offset. The tricky issue is that for a compacted topic, the underlying
log segment could have changed between two consecutive fetch requests. We
could potentially make that case work, but the logic will be more
complicated.

Another thing is that it seems that the proposal only saves the metadata
overhead if there are low volume topics. If we use Jay's suggestion of
including 0 partitions in subsequent fetch requests, it seems that we could
get the metadata saving even if all topics have continuous traffic.

Thanks,

Jun
On Wed, Nov 22, 2017 at 1:14 PM, Colin McCabe <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB