Would I be correct in assuming that a Kafka cluster won't scale well to support lots (tens of millions) of topics? If I understand correctly, a node being added or removed would involve a leader election for each topic, which is a relatively expensive operation?
I didn't see any auto leader election for adding nodes. The data are still skewed on the old nodes. You have to force it by running script? On Wed, Nov 13, 2013 at 6:41 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
Zookeeper will not be the only problem. The first is that each topic is a directory on the file system. Each of those is going to have files inside it. This is going to be fairly overwhelming for the file system. Also I can not speak for the internals but there may be cases where this many topics allocates a big array or some other non-optimal behaviour.
Like a RDBMS with this many tables one might ask, why? Isn't there a way to design the system multi-tennent where so many physical topics are not needed? On Wed, Nov 13, 2013 at 9:41 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
Thanks for the replies. I don't think Kafka quite fits our use case, unfortunately. To abstractly answer Edward's question: in a system with lots of users, we were considering having a topic per user (such that an individual user can connect from a number of endpoints and receive events, including events that were sent while the user was disconnected - persisting the events to disk and using offsets means we don't have to track which events each individual endpoint has received).
On 14 November 2013 04:38, Edward Capriolo <[EMAIL PROTECTED]> wrote: Bitroot - http://bitroot.com
If I understand your comment correctly you are trying to use kafka topics as per-endpoint message queues.
I may be mistaken, but to me Kafka seems does not really seem like a good match for that. For such a purpose you will eventually want something that is not actually a queue - a means to perform state compression, event-type-based and priority-based dequeue and other operations which are not appropriate to a throughput/FIFO oriented system. On Nov 14, 2013, at 6:18 AM, Joe Freeman <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext