Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Topic messages with partitions=1 stored on multiple brokers


Copy link to this message
-
Re: Topic messages with partitions=1 stored on multiple brokers
It may something with your configuration. Try it with a simpler setup of one broker, see that both topics appear on the broker, and build on your understanding and experimentation with that setup.

Philip

On Sep 20, 2013, at 11:06 AM, Suchi Amalapurapu <[EMAIL PROTECTED]> wrote:

> Thx for your replies. Can u please clarify this further for me?
>
> In 0.7, if the partition count is per broker, shouldn't each broker have a
> log directory for every topic?
> I don't see that though... Only some topics have logs in different brokers.
>
> How is the consumer expected to handle this? My start sequence is as follows
> - start zookeeper
> - start producer
> - start broker1
> - start broker2
> - send message to topic1
> - send message to topic 2
>
> log directory for topic1 is created on broker1 and broker2
> log directory for topic2 is created only on broker1
>
> The consumer(kafka-storm spout) with a parallelism of one should ideally
> discover the brokers via zookeeper and in this case should read the
> messages in the correct sequence from both?
> Suchi
>
>
> On Fri, Sep 20, 2013 at 11:09 PM, Suchi Amalapurapu <[EMAIL PROTECTED]>wrote:
>
>> Yes I understand that. I am letting the producer/consumer use zookeeper to
>> discover brokers.
>> I can clearly see in the logs(brokers) that both the brokers create a new
>> topic log for the same topic.
>>
>> The brokers are in different availability zones. Does that matter?
>> Suchi
>>
>>
>> On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>>
>>> Seems to me you are confusing partitions and brokers. Partition count has
>>> nothing to do with the number of brokers to which a message a sent -- just
>>> the number of partitions into which that message will be split when it
>>> gets
>>> to a broker.
>>>
>>> You need to explicitly set the destination brokers in the Producer, not
>>> via
>>> setting "partition" count.
>>>
>>> Philip
>>>
>>>
>>> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu <[EMAIL PROTECTED]
>>>> wrote:
>>>
>>>> My current setup has 2 brokers, multiple topics, each topic with 1
>>>> partition, kafka 0.7.1
>>>> Some topics are logged onto multiple brokers inspite of partitions set
>>> to
>>>> 1. Kafka 0.7.1 does not have replication turned on.
>>>> Why are the messages being logged onto multiple brokers when the number
>>> of
>>>> partitions is set to 1?
>>>> I have a consumer with parallelism set to 1 and its missing some of
>>> these
>>>> messages.
>>>> Suchi
>>
>>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB