Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Dilemma - ZK consumer problems - upgrade to 0.8?

Copy link to this message
Re: Dilemma - ZK consumer problems - upgrade to 0.8?
Which version of ZK are you using ?
On Fri, Jun 7, 2013 at 12:15 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> Hey guys,
> I sincerely apologize if this has been covered before, I haven't quite
> found a similar situation.
> We are using Kafka 0.7.2 in production, and we are using the ZK high level
> Scala consumer.   However, we find the ZK consumer very unstable.  It would
> work for one or two weeks, then suddenly it would complain about ZK nodes
> disappearing, and one consumer would die, then another, then another, until
> our pipeline is no longer pulling any data.   There are multiple
> NullPointerExceptions, and other problems.    We can restart it, but it
> does not stay up predictably.
> On the other hand, I have a simple app which I wrote using the simple
> consumer to mirror select partitions (will blog about this later) and it
> just works flawlessly.
> So we are faced with a dilemma to get back on track:
> 1)  Use SimpleConsumer, and write our own balancing code  (but honestly our
> boxes almost never go down, compared to the rate of ZK mishaps)
> 2)  Upgrade to Kafka 0.8 and hope that that resolves the issue.
> There seem to be so many improvements in 0.8 that that seems to be the
> biggest win long-term, so I am wondering if people can comment on:
> - has anyone tried using 0.8 in production?  Is it stable yet?
> - How much more stable is the ZK consumer in 0.8?
> - will it be possible to change the offset in the 0.8 consumer?  That was
> the other reason why we wanted to move to SimpleConsumer.
> thanks,
> Evan
> --
> --
> Evan Chan
> Staff Engineer
> <http://www.ooyala.com/>
> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><
> http://www.twitter.com/ooyala>