Newly added partitions in 0.8.1.1 dont see data immediately
I used the add partition functionality in create-topics to alter a previous
topic and increase the partitions. I noticed that after the new partitions
were added, they dont receive data immediately from the producer unless a
new producer is started up or the old producer is restarted.
Here is the sequence of steps:
1) start cluster with 2 brokers.
2) create topic with 2 partitions and replication factor 2.
3) start 4 consumers in a group. 2 consumers are redundant.
4) Start the performance producer to send messages.
5) Now alter the topic to have 4 partitions.
6) zookeeper immediately shows the new partitions
7) On watching the partitions through Kafka Offset Monitor, I dont see any
messages going into the 2 new partitions.
8) However, if i start a new performance producer or restart the one in 4)
I see that all the partitions are written to.
Why does 7) happen ? Is this again related to the sticky partitioning
behavior change in 0.8.11 ?
My producer.properties has topic.metadata.refresh.interval.ms=1000