Okay I did some assessment of use cases we have which aren't using the
default offset storage API and came up with one generalization. I would
like to propose--add a generic metadata field to the offset api on a
per-partition basis. So that would leave us with the following:

OffsetCommitRequest => ConsumerGroup [TopicName [Partition Offset Metadata]]

OffsetFetchResponse => [TopicName [Partition Offset Metadata ErrorCode]]

  Metadata => string

If you want to store a reference to any associated state (say an HDFS file
name) so that if the consumption fails over the new consumer can start up
with the same state, this would be a place to store that. It would not be
intended to support large stuff (we could enforce a 1k limit or something,
just something small or a reference on where to find the state (say a file


On Mon, Dec 17, 2012 at 10:45 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB