The following are some comments on the SimpleConsumer wiki.

1. PartitionMetadata.leader() can return null if the new leader is not
elected yet. We need to handle that.
2. When using FetchRequestBuilder, it's important NOT to set replicaId
since this is only meant to be used by fetchers in the follower replicas.
Setting replicaId incorrectly will cause the broker to behave strangely. We
didn't do that in the code, but it would be useful to add a comment to
highlight this.
3. The following code doesn't match the comment. The comment says resetting
to the last offset, but the code resets to the first offset.

                    // We asked for an invalid offset. For simple case
ask for the last element to reset                    readOffset =
getLastOffset(consumer,a_topic, a_partition,
kafka.api.OffsetRequest.EarliestTime(), clientName);

4. It seems that we can combine findNewLeader() and fineLeader() into one
method. In particular, the port of the new leader may not be the same as
the old one.
5. There is one more surprise when iterating messages in a messageSet
retuned in a fetch response. In general, if the fetch offset is X, you may
get messages with offsets less than X in the returned messageSet. This is
because in 0.8, each message in a compressed unit has its own offset. So,
if a compressed messageSet contains 10 messages with offsets from 0 to 9
and the client wants to fetch from offset 5, the broker is going to return
all 10 messages. So, it's the responsibility of the client to skip the
first 5 messages with offsets from 0 to 4 and only consume messages with
offset 5 and above. Otherwise, the client may get duplicates. This is
handled in the high level consumer and we need to handle that in
SimpleConsumer as well.



On Fri, Mar 29, 2013 at 8:28 AM, Chris Curtin <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB