Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Newly added partitions in 0.8.1.1 dont see data immediately


Copy link to this message
-
Newly added partitions in 0.8.1.1 dont see data immediately
Hi,

I used the add partition functionality in create-topics to alter a previous
topic and increase the partitions. I noticed that after the new partitions
were added, they dont receive data immediately from the producer unless a
new producer is started up or the old producer is restarted.
Here is the sequence of steps:
1) start cluster with 2 brokers.
2) create topic with 2 partitions and replication factor 2.
3) start 4 consumers in a group. 2 consumers are redundant.
4) Start the performance producer to send messages.
5) Now alter the topic to have 4 partitions.
6) zookeeper immediately shows the new partitions
7) On watching the partitions through Kafka Offset Monitor, I dont see any
messages going into the 2 new partitions.
8) However, if i start a new performance producer or restart the one in 4)
I see that all the partitions are written to.

Why does 7) happen ? Is this again related to the sticky partitioning
behavior change in 0.8.11 ?
My producer.properties has topic.metadata.refresh.interval.ms=1000

Thanks,
Prakash

 
+
Jun Rao 2014-06-17, 03:38
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB