Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> [jira] [Resolved] (KAFKA-278) Issues partitioning a new topic


Copy link to this message
-
[jira] [Resolved] (KAFKA-278) Issues partitioning a new topic

     [ https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Kreps resolved KAFKA-278.
-----------------------------

    Resolution: Fixed

This should be fixed in 0.8 where we handle partitioning properly.
                
> Issues partitioning a new topic
> -------------------------------
>
>                 Key: KAFKA-278
>                 URL: https://issues.apache.org/jira/browse/KAFKA-278
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Matt
>            Priority: Minor
>         Attachments: bootstrap_new_brokers.patch
>
>
> There are two cases where correct partitioning fails for a new topic.
> Case 1: Topic exists on current Kafka cluster. A new broker is added to the cluster. The new broker will never host partitions for the existing topic.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages for a topic to the cluster.
> 3) Add a new broker to the cluster.
> 4) New broker will never see the existing topic.
> Case 2: Topic does not exist on current Kafka cluster. Producer sends messages to a new topic that did not previously exist in the cluster. If, during the producer session, one or more partitions are not created on a broker, the broker will never host those partitions.
> To reproduce:
> 1) Create a cluster of brokers along with a ZooKeeper ensemble.
> 2) Send messages to a new topic.
> 3) Shut down the producer before the topic is created on at least one broker.
> 4) The broker that did not allocate the topic will never host the topic.
> My guess(!) here is that when a new producer is created, it gets a list of topics and partitions based on the current state of the brokers in the cluster. Since some brokers are missing the topic, the producer will never send messages to that broker and partitions will never be created.
> Work around:
> Manually create the topic/partition directories in the kafka logs directory and reboot kafka. It will register the topic/partitions in ZooKeeper.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB