Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # dev - zookeeper partition data


Copy link to this message
-
zookeeper partition data
Jay Kreps 2013-01-17, 03:48
Why is the node<=>partition mapping stored as strings?

get /brokers/topics/my-topic
{ "0": ["0"] }

This is a bug, right? Partitions and node ids are ints.

Also, this structure is not very extensible. If you add anything to this
map that is not a partition-node mapping it would break. I think the way
this should have been done is

{
  "partitions:{0:[1,3,2], 1:[3,1,7], ...}
}

Then we could add more attributes to the map:

{
  "partitions:{0:[1,3,2], 1:[3,1,7], ...}
  "settings":{
     "log.segment.bytes":1234234,
      ...
   }
}

I think our zookeeper data structures are another "contract" it would be
good to document and review these before the 0.8 release. If we want to
change something now it is really easy but once it is out there all changes
will need to be backwards compatible since they are persistent and can be
shared by all brokers (so rolling upgrade will be difficult).

-Jay

 
+
Jun Rao 2013-01-17, 15:59