Millions of messages per day (with each message being few bytes) is not
really 'Big Data'. Kafka has been tested for a million message per second.
The answer to all your question IMO is "It depends".
You can start with a single instance (Single machine installation). Let
your producer send messages. Keep one broker. Increase to N brokers. When
you touch the upper limit add a server and repeat all the stuff.
Bench marking and scalability are aspects which you should try on your own
by playing with Kafka. Every use case is different. So performance metric
of one is not a global answer.
For your question on Topic or Queue, please read something about
Distributed Computing Pub/Sub, Message Queue's and other patterns which are
generic concepts and has nothing to do with Kafka. It again depends on your
Please read as to what topics in Kafka are? If you just go through the
definition of topics you would yourself answer your question within a
Replications and all would be next steps once you are done with a single
running instance of Kafka. So go ahead and get your hands dirty. You will
love Kafka :)
And yes, the most important thing: Please read the documentation first (bit
of theory) and then dive. There is no silver bullet.
On Mon, Jul 22, 2013 at 4:27 PM, <[EMAIL PROTECTED]> wrote:
> I am planning to use Apache Kafka 0.8 to handle millions of messages per
> day. Now I need to form the environment, like
> (i) How many Topics to be created?
> (ii) How many partitions/replications to be created?
> (iii) How many Brokers to be created?
> (iv) How many consumer instances in consumer group?
> (v) Topic or Queue? If topic whether we need to create multiple group Id
> as supposed to single one?
> How we can go about it? Please clarify.
> Thanks & Regards,
> Please do not print this email unless it is absolutely necessary.
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments.
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> The company accepts no liability for any damage caused by any virus
> transmitted by this email.