Which version of ZK are you using ?
On Fri, Jun 7, 2013 at 12:15 PM, Evan Chan <[EMAIL PROTECTED]> wrote:
> Hey guys,
> I sincerely apologize if this has been covered before, I haven't quite
> found a similar situation.
> We are using Kafka 0.7.2 in production, and we are using the ZK high level
> Scala consumer. However, we find the ZK consumer very unstable. It would
> work for one or two weeks, then suddenly it would complain about ZK nodes
> disappearing, and one consumer would die, then another, then another, until
> our pipeline is no longer pulling any data. There are multiple
> NullPointerExceptions, and other problems. We can restart it, but it
> does not stay up predictably.
> On the other hand, I have a simple app which I wrote using the simple
> consumer to mirror select partitions (will blog about this later) and it
> just works flawlessly.
> So we are faced with a dilemma to get back on track:
> 1) Use SimpleConsumer, and write our own balancing code (but honestly our
> boxes almost never go down, compared to the rate of ZK mishaps)
> 2) Upgrade to Kafka 0.8 and hope that that resolves the issue.
> There seem to be so many improvements in 0.8 that that seems to be the
> biggest win long-term, so I am wondering if people can comment on:
> - has anyone tried using 0.8 in production? Is it stable yet?
> - How much more stable is the ZK consumer in 0.8?
> - will it be possible to change the offset in the 0.8 consumer? That was
> the other reason why we wanted to move to SimpleConsumer.
> Evan Chan
> Staff Engineer
> [EMAIL PROTECTED] |