Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Offset not written back to zookeeper


Copy link to this message
-
Re: Offset not written back to zookeeper
How is that modified version different from 0.7.1? Did you take a thread
dump and see what the offset commit thread was doing?

Thanks,

Jun

On Mon, Oct 29, 2012 at 7:34 AM, Raimar Falke <[EMAIL PROTECTED]>wrote:

> Hello
>
> We are using a modified version of kafka 0.7.1 (
> https://github.com/optivo-org/kafka)
> and had at least one case where the consumers of the group got events but
> the offsets
> were not written back to zookeeper. The consumers stopped updating the
> offsets after
> a release where the consumers of the group are each individually is
> restarted. Here
> a lot of rebalancing was taking place. After a further restart of all
> consumers the
> events were processed a second time. No related warning or error message
> could be
> found.
>
> Is this a known problem? What could be the cause for this?
>
> Tracing the code paths it seems weird, that
> ZookeeperConsumerConnector.commitOffsets()
> does not issue a warning if no offsets were written. Is it possible that a
> rebalancing
> was not finished and as such events were processed but without an
> ownership no updates
> take place?
>
>     Raimar
>
> optivo GmbH
> Wallstraße 16
> 10179 Berlin
>
> Telefon: 030 / 76 80 78 163
> Fax:     030 / 76 80 78 499
>
> E-Mail:  [EMAIL PROTECTED]
> Website: http://www.optivo.de
>
> Handelsregister: HRB Berlin 88738
> Geschäftsführer: Peter Romianowski, Ulf Richter
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB