Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Dilemma - ZK consumer woes - upgrade to 0.8?


Copy link to this message
-
Re: Dilemma - ZK consumer woes - upgrade to 0.8?
Jun Rao 2013-06-08, 04:05
Which version of ZK are you using?

Thanks,

Jun
On Fri, Jun 7, 2013 at 12:20 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> [ Sorry if this mail is duplicated, this is my fourth try sending this
> message]
>
> Hey guys,
>
> I sincerely apologize if this has been covered before, I haven't quite
> found a similar situation.
>
> We are using Kafka 0.7.2 in production, and we are using the ZK high level
> Scala consumer.   However, we find the ZK consumer very unstable.  It would
> work for one or two weeks, then suddenly it would complain about ZK nodes
> disappearing, and one consumer would die, then another, then another, until
> our pipeline is no longer pulling any data.   There are multiple
> NullPointerExceptions, and other problems.    We can restart it, but it
> does not stay up predictably.
>
> On the other hand, I have a simple app which I wrote using the simple
> consumer to mirror select partitions (will blog about this later) and it
> just works flawlessly.
>
> So we are faced with a dilemma to get back on track:
> 1)  Use SimpleConsumer, and write our own balancing code  (but honestly our
> boxes almost never go down, compared to the rate of ZK mishaps)
> 2)  Upgrade to Kafka 0.8 and hope that that resolves the issue.
>
> There seem to be so many improvements in 0.8 that that seems to be the
> biggest win long-term, so I am wondering if people can comment on:
> - has anyone tried using 0.8 in production?  Is it stable yet?
> - How much more stable is the ZK consumer in 0.8?
> - will it be possible to change the offset in the 0.8 consumer?  That was
> the other reason why we wanted to move to SimpleConsumer.
>
> thanks,
> Evan
>