Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> [jira] [Commented] (KAFKA-554) Move all per-topic configuration into ZK and add to the CreateTopicCommand

Jay Kreps 2013-03-05, 01:04
Neha Narkhede 2013-03-06, 20:48
Copy link to this message
[jira] [Commented] (KAFKA-554) Move all per-topic configuration into ZK and add to the CreateTopicCommand

    [ https://issues.apache.org/jira/browse/KAFKA-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597385#comment-13597385 ]

Neha Narkhede commented on KAFKA-554:

- It seems like the map and sequence case inserts an extra "," for the last element as well. This will cause invalid json
- Does it make sense to start the health check at the end after the request handlers and replica manager has started up ?
- In shutdown(), you are closing zkclient before controller, replica manager, log manager which use zkclient. I think we should close it at the very end
> Move all per-topic configuration into ZK and add to the CreateTopicCommand
> --------------------------------------------------------------------------
>                 Key: KAFKA-554
>                 URL: https://issues.apache.org/jira/browse/KAFKA-554
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Jay Kreps
>              Labels: project
>             Fix For: 0.8.1
>         Attachments: KAFKA-554-v1.patch, KAFKA-554-v2.patch, KAFKA-554-v3.patch
> We have a number of per-topic configurations that control message retention and flush interval. Here is the list of properties I find in KafkaConfig that appear to be per-topic:
>   topic.log.file.size
>   topic.log.roll.hours
>   topic.log.retention.hours
>   topic.log.retention.size
>   topic.flush.intervals.ms
> Currently we specify these in server.properties. This is not a good solution as it requires a rolling bounce of the cluster to make a change, which just doesn't scale to having hundreds of topics. Also the map encoded in a CSV string is kind of hacky.
> We should move these into ZK in some kind of JSON blob that allows easily adding new per-topic configs and we should remove these from server.properties.
> It would be good to start with a wiki design and get consensus on that first.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Jun Rao 2013-03-12, 17:38