Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: a few questions from high level consumer documentation.


Copy link to this message
-
RE: a few questions from high level consumer documentation.


> -----Original Message-----
> From: Chris Curtin [mailto:[EMAIL PROTECTED]]

> > 1 When you say the iterator may block, do you mean hasNext() may block?
> >
>
> Yes.

Is this due to a potential non-blocking fetch (broker/zookeeper returns an
empty block if offset is current)?  Yet this blocks the network call of the
consumer iterator, do I have that right?  Are there other reasons it could
block?  Like the call fails and a backup call is made?

> > b.      For client crash, what can client do to avoid duplicate messages
> > when restarted? What I can think of is to read last message from log
> > file and ignore the first few received duplicate messages until
> > receiving the last read message. But is it possible for client to read
log file
> directly?
> >
>
> If you can't tolerate the possibility of duplicates you need to look at
the
> Simple Consumer example, There you control the offset storage.

Do you have example code that manages only once, even when a consumer for a
given partition goes away?

What does happen with rebalancing when a consumer goes away?  Is this
behavior of the high-level consumer group?  Is there a way to supply one's
own simple consumer with only once, within a consumer group that rebalances?

What happens if a producer goes away?

thanks much,
rob

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB