Kafka, mail # dev - Re: Offset commit api - 2012-12-20, 21:04
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
+
Jun Rao 2012-12-18, 16:06
+
Jay Kreps 2012-12-18, 16:23
Copy link to this message
-
Re: Offset commit api
Okay I did some assessment of use cases we have which aren't using the
default offset storage API and came up with one generalization. I would
like to propose--add a generic metadata field to the offset api on a
per-partition basis. So that would leave us with the following:

OffsetCommitRequest => ConsumerGroup [TopicName [Partition Offset Metadata]]

OffsetFetchResponse => [TopicName [Partition Offset Metadata ErrorCode]]

  Metadata => string

If you want to store a reference to any associated state (say an HDFS file
name) so that if the consumption fails over the new consumer can start up
with the same state, this would be a place to store that. It would not be
intended to support large stuff (we could enforce a 1k limit or something,
just something small or a reference on where to find the state (say a file
name).

Objections?

-Jay
On Mon, Dec 17, 2012 at 10:45 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
 
+
David Arthur 2012-12-20, 21:31
+
Jay Kreps 2012-12-20, 22:04
+
Jay Kreps 2012-12-20, 22:05
+
David Arthur 2012-12-20, 22:09
+
Milind Parikh 2012-12-20, 22:15
+
Jay Kreps 2012-12-20, 22:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB