Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Dilemma - ZK consumer woes - upgrade to 0.8?


Copy link to this message
-
Dilemma - ZK consumer woes - upgrade to 0.8?
[ Sorry if this mail is duplicated, this is my fourth try sending this
message]

Hey guys,

I sincerely apologize if this has been covered before, I haven't quite
found a similar situation.

We are using Kafka 0.7.2 in production, and we are using the ZK high level
Scala consumer.   However, we find the ZK consumer very unstable.  It would
work for one or two weeks, then suddenly it would complain about ZK nodes
disappearing, and one consumer would die, then another, then another, until
our pipeline is no longer pulling any data.   There are multiple
NullPointerExceptions, and other problems.    We can restart it, but it
does not stay up predictably.

On the other hand, I have a simple app which I wrote using the simple
consumer to mirror select partitions (will blog about this later) and it
just works flawlessly.

So we are faced with a dilemma to get back on track:
1)  Use SimpleConsumer, and write our own balancing code  (but honestly our
boxes almost never go down, compared to the rate of ZK mishaps)
2)  Upgrade to Kafka 0.8 and hope that that resolves the issue.

There seem to be so many improvements in 0.8 that that seems to be the
biggest win long-term, so I am wondering if people can comment on:
- has anyone tried using 0.8 in production?  Is it stable yet?
- How much more stable is the ZK consumer in 0.8?
- will it be possible to change the offset in the 0.8 consumer?  That was
the other reason why we wanted to move to SimpleConsumer.

thanks,
Evan

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB