Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Proposal: fixed release schedule


Copy link to this message
-
Re: Proposal: fixed release schedule
Chris Burroughs 2011-11-15, 00:39
I agree this sounds reasonable.  In particular KAFKA-172, KAFKA-134, and
especially KAFKA-169 are examples of the kind of things I think are
worthwhile to get shipped into releases sooner rather than latter.

Thanks everyone.

On 11/14/2011 12:49 AM, Jay Kreps wrote:
> So the proposal I heard was:
>
>    - We create a branch for replication and begin development work on it
>    there.
>    - We leave trunk for incremental fixes and non-replication feature
>    development.
>    - We do monthly releases off trunk with whatever we have.
>    - We will number accordingly, so if we only have some small fixes then
>    the release will just be 0.7.1 or 0.7.01 or whatever; replication or larger
>    features will bump us up to 0.8.
>
> That seems reasonable to me. What this does mean, though is that LinkedIn
> may or may not end up running the releases in production before they go
> out, so this does put more testing/validation burden on the open source
> process.
>
> -Jay
>
> On Sun, Nov 13, 2011 at 6:47 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Hi, Chris,
>>
>> Yes, the changes related to replication are significant. It's actually a
>> bit hard to keep what we have now as the degenerated case with R=1.
>> Currently, the partions are numbered locally within each broker. This is
>> problematic with replication since a partition can have multiple replicas
>> that physically exist on multiple brokers. So, in our replication design,
>> partitions will be numbered globally within the cluster. This implies
>> potential on disk layout changes (see kafka-47) and wire protocol changes.
>>
>> You brought up a good question on migration path from 0.7 to 0.8. Since
>> replication changes are significant, it's going to be hard to make those
>> changes while keeping backward compatibility. My current feeling is that we
>> will make 0.8 a non-backward compatible release. To migrate to 0.8,
>> one possibility is to first create a new cluster of Kafka brokers using 0.8
>> (can potentially overlay on existing hardware). Then upgrade all producers
>> and point them to the new brokers. Drain all old consumers. Upgrade all
>> consumers and point them to the new brokers.
>>
>> What do people feel about this?
>>
>> Thanks,
>>
>> Jun
>>
>> On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs
>> <[EMAIL PROTECTED]>wrote:
>>
>>> On 11/11/2011 06:01 PM, Jay Kreps wrote:
>>>> 2. Our current focus for linkedin folks was going to be on replication
>>>>    which will be a really disruptive set of features that need to be
>>> released
>>>>    together.
>>>
>>> I think there are some differing assumptions on the impact of
>>> replication.  My assumption has been that replication would be a new
>>> feature that was 'optional' in that R=1 would not be significantly from
>>> what we are doing now.  Thus replication work could continue in trunk --
>>> and even be present in releases -- without being disruptive, the lack
>>> change would just be to allow and document values of R > 1.
>>>
>>> However, both Jay and Jun's description of the work make it sound like
>>> more of a backwards incompatible no-rolling-restart every
>>> broker/consumer/producer needs to be upgraded at the same time sort of
>>> change.
>>>
>>
>