On Wed, Nov 22, 2017, at 14:09, Jun Rao wrote:

Hmm.  So the way I was thinking about it was, with an incremental fetch
request, for each partition:

1a. the leader consults its cache to find the offset it needs to use for
the fetch request
2a. the leader performs a lookup to translate the offset to a file index
3a. the leader reads the data from the file

In contrast, with a full fetch request, for each partition:

1b. the leader looks at the FetchRequest to find the offset it needs to
use for the fetch request
2b. the leader performs a lookup to translate the offset to a file index
3b. the leader reads the data from the file

It seems like there is only one offset index lookup in both cases?  The
key point is that the cache in step #1a is not stored on disk.  Or maybe
I'm missing something here.

best,
Colin
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB