Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Kafka Cluster issue..


Copy link to this message
-
Re: Kafka Cluster issue..
Jun Rao 2013-02-11, 17:15
There is a known bug in Kafka 0.7.x. Basically, for a new topic, we
bootstrap using all existing brokers. However, if a topic already exists on
some brokers, we never bootstrap again, which means new brokers will be
ignored. For now, you have to manually create the topic on the new brokers
(e.g., by sending some data to the new broker directly).

See the discussion in

http://apache.markmail.org/thread/c7tdalfketpusqkg<https://email.corp.linkedin.com/owa/redir.aspx?C=vZZC9djRrEur-lWAgOQM7PtQdxMJ3M8IQEdO3N7Snpy8CUDf2IYTRBuzdFKt-90dlUl7k3E8H8k.&URL=http%3a%2f%2fapache.markmail.org%2fthread%2fc7tdalfketpusqkg>

As for configs, you can take a look at
https://cwiki.apache.org/KAFKA/operations.html

Thanks,

Jun

On Fri, Feb 8, 2013 at 7:28 AM, Yogesh G <[EMAIL PROTECTED]> wrote:

> Hi
>
> I am using Kafka 0.7.2...the setup have two Kafka brokers running on two
> different machines both pointing to a single Zookeper server which is on
> one of the machines hosting Kafka....Now if I run the Producer, either one
> provided in getting started guide or custom one, although it states that
> Creating async producer for broker 1 and broker 2....but the entire load is
> being processed by single Kafka broker...the second instance have no impact
> and nor does it creates any logs....I have num.partition=6 for both
> brokers....
>
> I would like to understand that wouldn't Zookeeper discover the two brokers
> and distribute the load.....am I missing anything here....
>
> Also for supporting a load of 2 million messages (say 200KB each) what is
> kind of topology and configuration is recommended...
>
> Thanks,
>