Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Proposal: fixed release schedule


Copy link to this message
-
Re: Proposal: fixed release schedule
I agree this sounds reasonable.  In particular KAFKA-172, KAFKA-134, and
especially KAFKA-169 are examples of the kind of things I think are
worthwhile to get shipped into releases sooner rather than latter.

Thanks everyone.

On 11/14/2011 12:49 AM, Jay Kreps wrote:
> So the proposal I heard was:
>
>    - We create a branch for replication and begin development work on it
>    there.
>    - We leave trunk for incremental fixes and non-replication feature
>    development.
>    - We do monthly releases off trunk with whatever we have.
>    - We will number accordingly, so if we only have some small fixes then
>    the release will just be 0.7.1 or 0.7.01 or whatever; replication or larger
>    features will bump us up to 0.8.
>
> That seems reasonable to me. What this does mean, though is that LinkedIn
> may or may not end up running the releases in production before they go
> out, so this does put more testing/validation burden on the open source
> process.
>
> -Jay
>
> On Sun, Nov 13, 2011 at 6:47 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Hi, Chris,
>>
>> Yes, the changes related to replication are significant. It's actually a
>> bit hard to keep what we have now as the degenerated case with R=1.
>> Currently, the partions are numbered locally within each broker. This is
>> problematic with replication since a partition can have multiple replicas
>> that physically exist on multiple brokers. So, in our replication design,
>> partitions will be numbered globally within the cluster. This implies
>> potential on disk layout changes (see kafka-47) and wire protocol changes.
>>
>> You brought up a good question on migration path from 0.7 to 0.8. Since
>> replication changes are significant, it's going to be hard to make those
>> changes while keeping backward compatibility. My current feeling is that we
>> will make 0.8 a non-backward compatible release. To migrate to 0.8,
>> one possibility is to first create a new cluster of Kafka brokers using 0.8
>> (can potentially overlay on existing hardware). Then upgrade all producers
>> and point them to the new brokers. Drain all old consumers. Upgrade all
>> consumers and point them to the new brokers.
>>
>> What do people feel about this?
>>
>> Thanks,
>>
>> Jun
>>
>> On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs
>> <[EMAIL PROTECTED]>wrote:
>>
>>> On 11/11/2011 06:01 PM, Jay Kreps wrote:
>>>> 2. Our current focus for linkedin folks was going to be on replication
>>>>    which will be a really disruptive set of features that need to be
>>> released
>>>>    together.
>>>
>>> I think there are some differing assumptions on the impact of
>>> replication.  My assumption has been that replication would be a new
>>> feature that was 'optional' in that R=1 would not be significantly from
>>> what we are doing now.  Thus replication work could continue in trunk --
>>> and even be present in releases -- without being disruptive, the lack
>>> change would just be to allow and document values of R > 1.
>>>
>>> However, both Jay and Jun's description of the work make it sound like
>>> more of a backwards incompatible no-rolling-restart every
>>> broker/consumer/producer needs to be upgraded at the same time sort of
>>> change.
>>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB