Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Topic messages with partitions=1 stored on multiple brokers


Copy link to this message
-
Re: Topic messages with partitions=1 stored on multiple brokers
No, it does not matter, since that has no bearing on TCP connectivity.

Philip

On Sep 20, 2013, at 10:39 AM, Suchi Amalapurapu <[EMAIL PROTECTED]> wrote:

> Yes I understand that. I am letting the producer/consumer use zookeeper to
> discover brokers.
> I can clearly see in the logs(brokers) that both the brokers create a new
> topic log for the same topic.
>
> The brokers are in different availability zones. Does that matter?
> Suchi
>
>
> On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
>> Seems to me you are confusing partitions and brokers. Partition count has
>> nothing to do with the number of brokers to which a message a sent -- just
>> the number of partitions into which that message will be split when it gets
>> to a broker.
>>
>> You need to explicitly set the destination brokers in the Producer, not via
>> setting "partition" count.
>>
>> Philip
>>
>>
>> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu <[EMAIL PROTECTED]
>>> wrote:
>>
>>> My current setup has 2 brokers, multiple topics, each topic with 1
>>> partition, kafka 0.7.1
>>> Some topics are logged onto multiple brokers inspite of partitions set to
>>> 1. Kafka 0.7.1 does not have replication turned on.
>>> Why are the messages being logged onto multiple brokers when the number
>> of
>>> partitions is set to 1?
>>> I have a consumer with parallelism set to 1 and its missing some of these
>>> messages.
>>> Suchi
>>