How hard would it be to fix this issue, where we have a topic filter that matches multiple topics, for the load to be distributed over multiple threads, and over multiple consumers? For some reason, I had thought this issue was fixed in 0.8, but I guess not?
I am currently using a single partition, for multiple topics. I worry that it won't scale ultimately to only ever have one thread on one consumer doing all the work......We could move to multiple partitions, but for ordering reasons in some use cases, this is not always ideal.
Perhaps I can come up with some sort of dynamic topic sniffer, and have it evenly divide the available topics between the available consumers (and threads per consumer)! Is there a simple api within the kafka client code, for getting the list of topics?
Jason On Fri, Aug 30, 2013 at 11:41 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
Jason Rosenberg 2013-10-02, 20:29
Re: Num of streams for consumers using TopicFilter.
I filed this, to address the need for allowing parallelism when consuming multiple single-partition topics selected with a topic filter: https://issues.apache.org/jira/browse/KAFKA-1072 On Thu, Oct 3, 2013 at 10:56 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
Jason Rosenberg 2013-10-03, 15:12
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext