Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Topic messages with partitions=1 stored on multiple brokers


Copy link to this message
-
Re: Topic messages with partitions=1 stored on multiple brokers
It may something with your configuration. Try it with a simpler setup of one broker, see that both topics appear on the broker, and build on your understanding and experimentation with that setup.

Philip

On Sep 20, 2013, at 11:06 AM, Suchi Amalapurapu <[EMAIL PROTECTED]> wrote:

> Thx for your replies. Can u please clarify this further for me?
>
> In 0.7, if the partition count is per broker, shouldn't each broker have a
> log directory for every topic?
> I don't see that though... Only some topics have logs in different brokers.
>
> How is the consumer expected to handle this? My start sequence is as follows
> - start zookeeper
> - start producer
> - start broker1
> - start broker2
> - send message to topic1
> - send message to topic 2
>
> log directory for topic1 is created on broker1 and broker2
> log directory for topic2 is created only on broker1
>
> The consumer(kafka-storm spout) with a parallelism of one should ideally
> discover the brokers via zookeeper and in this case should read the
> messages in the correct sequence from both?
> Suchi
>
>
> On Fri, Sep 20, 2013 at 11:09 PM, Suchi Amalapurapu <[EMAIL PROTECTED]>wrote:
>
>> Yes I understand that. I am letting the producer/consumer use zookeeper to
>> discover brokers.
>> I can clearly see in the logs(brokers) that both the brokers create a new
>> topic log for the same topic.
>>
>> The brokers are in different availability zones. Does that matter?
>> Suchi
>>
>>
>> On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>>
>>> Seems to me you are confusing partitions and brokers. Partition count has
>>> nothing to do with the number of brokers to which a message a sent -- just
>>> the number of partitions into which that message will be split when it
>>> gets
>>> to a broker.
>>>
>>> You need to explicitly set the destination brokers in the Producer, not
>>> via
>>> setting "partition" count.
>>>
>>> Philip
>>>
>>>
>>> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu <[EMAIL PROTECTED]
>>>> wrote:
>>>
>>>> My current setup has 2 brokers, multiple topics, each topic with 1
>>>> partition, kafka 0.7.1
>>>> Some topics are logged onto multiple brokers inspite of partitions set
>>> to
>>>> 1. Kafka 0.7.1 does not have replication turned on.
>>>> Why are the messages being logged onto multiple brokers when the number
>>> of
>>>> partitions is set to 1?
>>>> I have a consumer with parallelism set to 1 and its missing some of
>>> these
>>>> messages.
>>>> Suchi
>>
>>