Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Offset not written back to zookeeper


Copy link to this message
-
Re: Offset not written back to zookeeper
Jun Rao 2012-10-30, 04:19
How is that modified version different from 0.7.1? Did you take a thread
dump and see what the offset commit thread was doing?

Thanks,

Jun

On Mon, Oct 29, 2012 at 7:34 AM, Raimar Falke <[EMAIL PROTECTED]>wrote:

> Hello
>
> We are using a modified version of kafka 0.7.1 (
> https://github.com/optivo-org/kafka)
> and had at least one case where the consumers of the group got events but
> the offsets
> were not written back to zookeeper. The consumers stopped updating the
> offsets after
> a release where the consumers of the group are each individually is
> restarted. Here
> a lot of rebalancing was taking place. After a further restart of all
> consumers the
> events were processed a second time. No related warning or error message
> could be
> found.
>
> Is this a known problem? What could be the cause for this?
>
> Tracing the code paths it seems weird, that
> ZookeeperConsumerConnector.commitOffsets()
> does not issue a warning if no offsets were written. Is it possible that a
> rebalancing
> was not finished and as such events were processed but without an
> ownership no updates
> take place?
>
>     Raimar
>
> optivo GmbH
> Wallstraße 16
> 10179 Berlin
>
> Telefon: 030 / 76 80 78 163
> Fax:     030 / 76 80 78 499
>
> E-Mail:  [EMAIL PROTECTED]
> Website: http://www.optivo.de
>
> Handelsregister: HRB Berlin 88738
> Geschäftsführer: Peter Romianowski, Ulf Richter
>