Kafka, mail # user - Re: Proper use of ConsumerConnector - 2012-12-20, 18:06
 Search Hadoop and all its subprojects:

Switch to Plain View
+
Tom Brown 2012-12-19, 18:49
+
Joel Koshy 2012-12-19, 19:16
+
永辉 赵 2012-12-20, 09:07
+
Neha Narkhede 2012-12-20, 16:56
Copy link to this message
-
Re: Proper use of ConsumerConnector
“unless you have a good reason to load balance and manage offsets manually”
An alternative to using simpleconsumer in this use case is to use the
zookeeper consumer connector and turn off auto commit. After your consumer
process is done processing a batch of messages you can all commitOffsets -
the main caveat to be aware of is that if your consumer processes batches
very fast you would write to zookeeper that often - so in fact setting an
autocommit interval and being willing to deal with duplicates is almost
equivalent. KAFKA-657 would help I think - since once that API is available
you can store your offsets anywhere you like.

Joel

 
+
Neha Narkhede 2012-12-20, 18:14
+
Tom Brown 2012-12-20, 19:32
+
Neha Narkhede 2012-12-20, 19:43
+
永辉 赵 2012-12-21, 10:56
+
Neha Narkhede 2012-12-21, 17:37
+
Tom Brown 2012-12-21, 17:47
+
Jun Rao 2012-12-21, 17:57
+
Yonghui Zhao 2012-12-21, 23:12
+
Tom Brown 2012-12-22, 01:44
+
Neha Narkhede 2012-12-23, 19:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB