With kafka 0.7 almost (*crosses fingers*) out I wanted to propose we
adopt a fixed release schedule for major releases instead of by feature.
 So that -- for example -- we set a freeze date for 0.8.0 that is n
weeks/months after 0.7.0 is completely out the door (something in the
2-4 month range perhaps).  Major changes could could live in branches or
(probably better) trunk with Crome-esque disabled feature flags.

This would have a few benifits:
 - Trunk will need to remain more stable, which encourages more testing
of trunk, RC releases, etc. in a virtuous cycle.
 - New features are available in a more predictable time frame to users
(less need for internal patching if something is Really Important).
 - Because it's clear when new features are coming down the road, there
is less temptation to add new features to bug fix releases.

The primary downside is that the this requires more discipline when
making larger changes (replication for example), and for this to work
well we will also need more comprehensive and reliable unit/system
tests.  I think in the long run those downsides will prove to be virtues.

To be clear, while I propose we stop making intentional changes at a
fixed date, the actual release still can't happen until we are happy
with the testing and bug-squashing that has gone into it (aka "when it's
ready").

Hi, Chris,

Thanks for bringing this up. My feeling is that fixed-time release is
probably more suitable for a more stable project. At this moment, we are
still developing major features in Kafka. The time it takes to add the
compression support is going to be quite different from adding the
replication support. So, for now, I'd rather do a major release with a
complete new feature, than at a fixed time interval. I agree that the
interval between 2 major releases shouldn't be significantly different. For
replication, I expect that it can be done in about 3-4 months once we get
started (should be in a week or two). We can still release bug fix releases
to patch critical bugs along the way.

Jun

On Fri, Nov 11, 2011 at 11:26 AM, Chris Burroughs <[EMAIL PROTECTED]
> wrote:

> With kafka 0.7 almost (*crosses fingers*) out I wanted to propose we
> adopt a fixed release schedule for major releases instead of by feature.
>  So that -- for example -- we set a freeze date for 0.8.0 that is n
> weeks/months after 0.7.0 is completely out the door (something in the
> 2-4 month range perhaps).  Major changes could could live in branches or
> (probably better) trunk with Crome-esque disabled feature flags.
>
> This would have a few benifits:
>  - Trunk will need to remain more stable, which encourages more testing
> of trunk, RC releases, etc. in a virtuous cycle.
>  - New features are available in a more predictable time frame to users
> (less need for internal patching if something is Really Important).
>  - Because it's clear when new features are coming down the road, there
> is less temptation to add new features to bug fix releases.
>
> The primary downside is that the this requires more discipline when
> making larger changes (replication for example), and for this to work
> well we will also need more comprehensive and reliable unit/system
> tests.  I think in the long run those downsides will prove to be virtues.
>
> To be clear, while I propose we stop making intentional changes at a
> fixed date, the actual release still can't happen until we are happy
> with the testing and bug-squashing that has gone into it (aka "when it's
> ready").
>

Jun,

You make valid points, but I still support Chris' viewpoint.  If you go a
long way and introduce a lot of big features in between releases you will
slow down upgrades of your early adopters.

Consider myself as a case study.  My team has been working on putting Kafka
into production for approximately 2 months.  In that time 0.7 has not
materialized so we are going with 0.6.  When we are stable we will have to
consider moving to 0.7 at some point.  But the size of the release in 0.7
makes me wary to do this and it will be a a major endeavor.

As an architect and engineering manager I prefer smaller, easier pieces of
functionality to consume.  It makes upgrading easier and less risky.

On Fri, Nov 11, 2011 at 12:00 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Hi, Chris,
>
> Thanks for bringing this up. My feeling is that fixed-time release is
> probably more suitable for a more stable project. At this moment, we are
> still developing major features in Kafka. The time it takes to add the
> compression support is going to be quite different from adding the
> replication support. So, for now, I'd rather do a major release with a
> complete new feature, than at a fixed time interval. I agree that the
> interval between 2 major releases shouldn't be significantly different. For
> replication, I expect that it can be done in about 3-4 months once we get
> started (should be in a week or two). We can still release bug fix releases
> to patch critical bugs along the way.
>
> Jun
>
> On Fri, Nov 11, 2011 at 11:26 AM, Chris Burroughs <
> [EMAIL PROTECTED]
> > wrote:
>
> > With kafka 0.7 almost (*crosses fingers*) out I wanted to propose we
> > adopt a fixed release schedule for major releases instead of by feature.
> >  So that -- for example -- we set a freeze date for 0.8.0 that is n
> > weeks/months after 0.7.0 is completely out the door (something in the
> > 2-4 month range perhaps).  Major changes could could live in branches or
> > (probably better) trunk with Crome-esque disabled feature flags.
> >
> > This would have a few benifits:
> >  - Trunk will need to remain more stable, which encourages more testing
> > of trunk, RC releases, etc. in a virtuous cycle.
> >  - New features are available in a more predictable time frame to users
> > (less need for internal patching if something is Really Important).
> >  - Because it's clear when new features are coming down the road, there
> > is less temptation to add new features to bug fix releases.
> >
> > The primary downside is that the this requires more discipline when
> > making larger changes (replication for example), and for this to work
> > well we will also need more comprehensive and reliable unit/system
> > tests.  I think in the long run those downsides will prove to be virtues.
> >
> > To be clear, while I propose we stop making intentional changes at a
> > fixed date, the actual release still can't happen until we are happy
> > with the testing and bug-squashing that has gone into it (aka "when it's
> > ready").
> >
>

Hi, Taylor,

Are you more worried about bug fixes? If so, we can have some fixed release
interval for bug fix releases.

Thanks,

Jun

No - just that big feature changes are more risky than smaller ones
and having smaller upgrades helps identify unexpected issues more
easily.

Having fixed time length releases reduces the tendency for releases to
grow larger.

Another benefit of frequent small releases is that you'll get good at
it so the process should become easier and more routine with time.
With big releases and long periods of real time in between this won't
be as likely to happen.

On Nov 11, 2011, at 1:08 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Hi, Taylor,
>
> Are you more worried about bug fixes? If so, we can have some fixed release
> interval for bug fix releases.
>
> Thanks,
>
> Jun

Taylor,

What's the release cycle that you are looking for?

Also, for big features like replication, it may be hard to decompose it
into multiple releases. We can probably stage it so that some of the more
advanced stuff (e.g., rebalance with new brokers) are released later.
However, even to get the basic replication work done requires significant
changes. Any suggests on how to do this better is welcomed.

Thanks,

Jun

On Fri, Nov 11, 2011 at 1:32 PM, Taylor Gautier <[EMAIL PROTECTED]> wrote:

> No - just that big feature changes are more risky than smaller ones
> and having smaller upgrades helps identify unexpected issues more
> easily.
>
> Having fixed time length releases reduces the tendency for releases to
> grow larger.
>
> Another benefit of frequent small releases is that you'll get good at
> it so the process should become easier and more routine with time.
> With big releases and long periods of real time in between this won't
> be as likely to happen.
>
>
>
> On Nov 11, 2011, at 1:08 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Hi, Taylor,
> >
> > Are you more worried about bug fixes? If so, we can have some fixed
> release
> > interval for bug fix releases.
> >
> > Thanks,
> >
> > Jun
>

I think maybe once per month might be reasonable? Or 6 weeks?  I don't
know exactly what exact timing would make sense.

I'm not necessarily suggesting to break up features into smaller
pieces and release them in parts. If a feature takes longer than a
release cycle to build then it just comes out in the cycle when it is
ready.

My point is that you will have less numbers of large features per
release with this strategy.

Of course it will be required for the feature owner of a large feature
to have a branch for development and be somewhat challenging to stay
abreast of the trunk changes but it's doable especially with git.

On Nov 11, 2011, at 1:50 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Taylor,
>
> What's the release cycle that you are looking for?
>
> Also, for big features like replication, it may be hard to decompose it
> into multiple releases. We can probably stage it so that some of the more
> advanced stuff (e.g., rebalance with new brokers) are released later.
> However, even to get the basic replication work done requires significant
> changes. Any suggests on how to do this better is welcomed.
>
> Thanks,
>
> Jun
>
> On Fri, Nov 11, 2011 at 1:32 PM, Taylor Gautier <[EMAIL PROTECTED]> wrote:
>
>> No - just that big feature changes are more risky than smaller ones
>> and having smaller upgrades helps identify unexpected issues more
>> easily.
>>
>> Having fixed time length releases reduces the tendency for releases to
>> grow larger.
>>
>> Another benefit of frequent small releases is that you'll get good at
>> it so the process should become easier and more routine with time.
>> With big releases and long periods of real time in between this won't
>> be as likely to happen.
>>
>>
>>
>> On Nov 11, 2011, at 1:08 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>
>>> Hi, Taylor,
>>>
>>> Are you more worried about bug fixes? If so, we can have some fixed
>> release
>>> interval for bug fix releases.
>>>
>>> Thanks,
>>>
>>> Jun
>>

In general I like this style of time-boxed monthly release planning. It
does make big disruptive changes harder but it makes small changes much
easier because they get out with regularity. It also gives a nice rhythm to
things.

There are a couple of potential issues for switching to this right now
though:

   1. To date our strategy has been to release at linkedin before doing an
   open source release. The advantage of this is that linkedin uses kafka very
   very heavily (literally thousands of producer and consumer processes,
   hundreds of topics, and a half dozen clusters separated by SLA). These
   rollouts take a lot of time and are really painful but I think they are
   valuable for other users because they ensure that by the time we do an open
   source release we are guaranteed pretty good quality. We have always found
   lots of issue in the course of each rollout. I am not sure how well we
   could manage this alignment with a more fixed schedule because the nature
   of things is somewhat stop and start--we rollout out on a few test
   clusters, then onto just some servers in a production cluster, then whole
   cluster, etc. Each time we find an issue we stop and debug it and then. On
   the other hand I am not sure how desirable it is to have kafka release
   schedule tied to linkedin since we want it to be a community driven project.
   2. Our current focus for linkedin folks was going to be on replication
   which will be a really disruptive set of features that need to be released
   together. Moving this to a branch would essentially move at least half of
   active development there too. Not sure if this is such a good idea. It
   might be better to just take a few months and get this worked out.

My personal opinion is that this would be a good change to make but we
should wait until after 0.8/replication goes out, at that point we should
be doing smaller iterations and the release alignment would be less of an
issue.

Any thoughts from others?

-Jay

On Fri, Nov 11, 2011 at 2:07 PM, Taylor Gautier <[EMAIL PROTECTED]> wrote:

> I think maybe once per month might be reasonable? Or 6 weeks?  I don't
> know exactly what exact timing would make sense.
>
> I'm not necessarily suggesting to break up features into smaller
> pieces and release them in parts. If a feature takes longer than a
> release cycle to build then it just comes out in the cycle when it is
> ready.
>
> My point is that you will have less numbers of large features per
> release with this strategy.
>
> Of course it will be required for the feature owner of a large feature
> to have a branch for development and be somewhat challenging to stay
> abreast of the trunk changes but it's doable especially with git.
>
> On Nov 11, 2011, at 1:50 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Taylor,
> >
> > What's the release cycle that you are looking for?
> >
> > Also, for big features like replication, it may be hard to decompose it
> > into multiple releases. We can probably stage it so that some of the more
> > advanced stuff (e.g., rebalance with new brokers) are released later.
> > However, even to get the basic replication work done requires significant
> > changes. Any suggests on how to do this better is welcomed.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Nov 11, 2011 at 1:32 PM, Taylor Gautier <[EMAIL PROTECTED]>
> wrote:
> >
> >> No - just that big feature changes are more risky than smaller ones
> >> and having smaller upgrades helps identify unexpected issues more
> >> easily.
> >>
> >> Having fixed time length releases reduces the tendency for releases to
> >> grow larger.
> >>
> >> Another benefit of frequent small releases is that you'll get good at
> >> it so the process should become easier and more routine with time.
> >> With big releases and long periods of real time in between this won't
> >> be as likely to happen.
> >>
> >>
> >>
> >> On Nov 11, 2011, at 1:08 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >>
> >>> Hi, Taylor,
> >>>
>

I think this is a good discussion. I agree with Taylor on the idea of doing
smaller 0.7.* releases before the big 0.8 one. This will encourage people
to keep adopting bug fixes and smaller features. Compare it to our wanting
to upgrade to zookeeper 3.3.4 before 3.4.0. That being said, I would think
we have to move to a new branch.

Thanks,
Neha

On Friday, November 11, 2011, Jay Kreps <[EMAIL PROTECTED]> wrote:
> In general I like this style of time-boxed monthly release planning. It
> does make big disruptive changes harder but it makes small changes much
> easier because they get out with regularity. It also gives a nice rhythm
to
> things.
>
> There are a couple of potential issues for switching to this right now
> though:
>
>   1. To date our strategy has been to release at linkedin before doing an
>   open source release. The advantage of this is that linkedin uses kafka
very
>   very heavily (literally thousands of producer and consumer processes,
>   hundreds of topics, and a half dozen clusters separated by SLA). These
>   rollouts take a lot of time and are really painful but I think they are
>   valuable for other users because they ensure that by the time we do an
open
>   source release we are guaranteed pretty good quality. We have always
found
>   lots of issue in the course of each rollout. I am not sure how well we
>   could manage this alignment with a more fixed schedule because the
nature
>   of things is somewhat stop and start--we rollout out on a few test
>   clusters, then onto just some servers in a production cluster, then
whole
>   cluster, etc. Each time we find an issue we stop and debug it and then.
On
>   the other hand I am not sure how desirable it is to have kafka release
>   schedule tied to linkedin since we want it to be a community driven
project.
>   2. Our current focus for linkedin folks was going to be on replication
>   which will be a really disruptive set of features that need to be
released
>   together. Moving this to a branch would essentially move at least half
of
>   active development there too. Not sure if this is such a good idea. It
>   might be better to just take a few months and get this worked out.
>
> My personal opinion is that this would be a good change to make but we
> should wait until after 0.8/replication goes out, at that point we should
> be doing smaller iterations and the release alignment would be less of an
> issue.
>
> Any thoughts from others?
>
> -Jay
>
> On Fri, Nov 11, 2011 at 2:07 PM, Taylor Gautier <[EMAIL PROTECTED]>
wrote:
>
>> I think maybe once per month might be reasonable? Or 6 weeks?  I don't
>> know exactly what exact timing would make sense.
>>
>> I'm not necessarily suggesting to break up features into smaller
>> pieces and release them in parts. If a feature takes longer than a
>> release cycle to build then it just comes out in the cycle when it is
>> ready.
>>
>> My point is that you will have less numbers of large features per
>> release with this strategy.
>>
>> Of course it will be required for the feature owner of a large feature
>> to have a branch for development and be somewhat challenging to stay
>> abreast of the trunk changes but it's doable especially with git.
>>
>> On Nov 11, 2011, at 1:50 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>
>> > Taylor,
>> >
>> > What's the release cycle that you are looking for?
>> >
>> > Also, for big features like replication, it may be hard to decompose it
>> > into multiple releases. We can probably stage it so that some of the
more
>> > advanced stuff (e.g., rebalance with new brokers) are released later.
>> > However, even to get the basic replication work done requires
significant
>> > changes. Any suggests on how to do this better is welcomed.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Fri, Nov 11, 2011 at 1:32 PM, Taylor Gautier <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> No - just that big feature changes are more risky than smaller ones
>> >> and having smaller upgrades helps identify unexpected issues more

On 11/11/2011 06:01 PM, Jay Kreps wrote:
> 2. Our current focus for linkedin folks was going to be on replication
>    which will be a really disruptive set of features that need to be released
>    together.

I think there are some differing assumptions on the impact of
replication.  My assumption has been that replication would be a new
feature that was 'optional' in that R=1 would not be significantly from
what we are doing now.  Thus replication work could continue in trunk --
and even be present in releases -- without being disruptive, the lack
change would just be to allow and document values of R > 1.

However, both Jay and Jun's description of the work make it sound like
more of a backwards incompatible no-rolling-restart every
broker/consumer/producer needs to be upgraded at the same time sort of
change.

Hi, Chris,

Yes, the changes related to replication are significant. It's actually a
bit hard to keep what we have now as the degenerated case with R=1.
Currently, the partions are numbered locally within each broker. This is
problematic with replication since a partition can have multiple replicas
that physically exist on multiple brokers. So, in our replication design,
partitions will be numbered globally within the cluster. This implies
potential on disk layout changes (see kafka-47) and wire protocol changes.

You brought up a good question on migration path from 0.7 to 0.8. Since
replication changes are significant, it's going to be hard to make those
changes while keeping backward compatibility. My current feeling is that we
will make 0.8 a non-backward compatible release. To migrate to 0.8,
one possibility is to first create a new cluster of Kafka brokers using 0.8
(can potentially overlay on existing hardware). Then upgrade all producers
and point them to the new brokers. Drain all old consumers. Upgrade all
consumers and point them to the new brokers.

What do people feel about this?

Thanks,

Jun

On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs
<[EMAIL PROTECTED]>wrote:

> On 11/11/2011 06:01 PM, Jay Kreps wrote:
> > 2. Our current focus for linkedin folks was going to be on replication
> >    which will be a really disruptive set of features that need to be
> released
> >    together.
>
> I think there are some differing assumptions on the impact of
> replication.  My assumption has been that replication would be a new
> feature that was 'optional' in that R=1 would not be significantly from
> what we are doing now.  Thus replication work could continue in trunk --
> and even be present in releases -- without being disruptive, the lack
> change would just be to allow and document values of R > 1.
>
> However, both Jay and Jun's description of the work make it sound like
> more of a backwards incompatible no-rolling-restart every
> broker/consumer/producer needs to be upgraded at the same time sort of
> change.
>

So the proposal I heard was:

   - We create a branch for replication and begin development work on it
   there.
   - We leave trunk for incremental fixes and non-replication feature
   development.
   - We do monthly releases off trunk with whatever we have.
   - We will number accordingly, so if we only have some small fixes then
   the release will just be 0.7.1 or 0.7.01 or whatever; replication or larger
   features will bump us up to 0.8.

That seems reasonable to me. What this does mean, though is that LinkedIn
may or may not end up running the releases in production before they go
out, so this does put more testing/validation burden on the open source
process.

-Jay

On Sun, Nov 13, 2011 at 6:47 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Hi, Chris,
>
> Yes, the changes related to replication are significant. It's actually a
> bit hard to keep what we have now as the degenerated case with R=1.
> Currently, the partions are numbered locally within each broker. This is
> problematic with replication since a partition can have multiple replicas
> that physically exist on multiple brokers. So, in our replication design,
> partitions will be numbered globally within the cluster. This implies
> potential on disk layout changes (see kafka-47) and wire protocol changes.
>
> You brought up a good question on migration path from 0.7 to 0.8. Since
> replication changes are significant, it's going to be hard to make those
> changes while keeping backward compatibility. My current feeling is that we
> will make 0.8 a non-backward compatible release. To migrate to 0.8,
> one possibility is to first create a new cluster of Kafka brokers using 0.8
> (can potentially overlay on existing hardware). Then upgrade all producers
> and point them to the new brokers. Drain all old consumers. Upgrade all
> consumers and point them to the new brokers.
>
> What do people feel about this?
>
> Thanks,
>
> Jun
>
> On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs
> <[EMAIL PROTECTED]>wrote:
>
> > On 11/11/2011 06:01 PM, Jay Kreps wrote:
> > > 2. Our current focus for linkedin folks was going to be on replication
> > >    which will be a really disruptive set of features that need to be
> > released
> > >    together.
> >
> > I think there are some differing assumptions on the impact of
> > replication.  My assumption has been that replication would be a new
> > feature that was 'optional' in that R=1 would not be significantly from
> > what we are doing now.  Thus replication work could continue in trunk --
> > and even be present in releases -- without being disruptive, the lack
> > change would just be to allow and document values of R > 1.
> >
> > However, both Jay and Jun's description of the work make it sound like
> > more of a backwards incompatible no-rolling-restart every
> > broker/consumer/producer needs to be upgraded at the same time sort of
> > change.
> >
>

This sounds good to me and restates what I've heard - I assume that the OSS
community will understand that new releases are bleeding edge,  it might
help to make this crystal clear on the releases page.  One thought I had
here is that x.0.0 might be considered not hammered on by LinkedIn, however
x.1.0 could be?

On Sun, Nov 13, 2011 at 9:49 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> So the proposal I heard was:
>
>   - We create a branch for replication and begin development work on it
>   there.
>   - We leave trunk for incremental fixes and non-replication feature
>   development.
>   - We do monthly releases off trunk with whatever we have.
>   - We will number accordingly, so if we only have some small fixes then
>   the release will just be 0.7.1 or 0.7.01 or whatever; replication or
> larger
>   features will bump us up to 0.8.
>
> That seems reasonable to me. What this does mean, though is that LinkedIn
> may or may not end up running the releases in production before they go
> out, so this does put more testing/validation burden on the open source
> process.
>
> -Jay
>
> On Sun, Nov 13, 2011 at 6:47 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Hi, Chris,
> >
> > Yes, the changes related to replication are significant. It's actually a
> > bit hard to keep what we have now as the degenerated case with R=1.
> > Currently, the partions are numbered locally within each broker. This is
> > problematic with replication since a partition can have multiple replicas
> > that physically exist on multiple brokers. So, in our replication design,
> > partitions will be numbered globally within the cluster. This implies
> > potential on disk layout changes (see kafka-47) and wire protocol
> changes.
> >
> > You brought up a good question on migration path from 0.7 to 0.8. Since
> > replication changes are significant, it's going to be hard to make those
> > changes while keeping backward compatibility. My current feeling is that
> we
> > will make 0.8 a non-backward compatible release. To migrate to 0.8,
> > one possibility is to first create a new cluster of Kafka brokers using
> 0.8
> > (can potentially overlay on existing hardware). Then upgrade all
> producers
> > and point them to the new brokers. Drain all old consumers. Upgrade all
> > consumers and point them to the new brokers.
> >
> > What do people feel about this?
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs
> > <[EMAIL PROTECTED]>wrote:
> >
> > > On 11/11/2011 06:01 PM, Jay Kreps wrote:
> > > > 2. Our current focus for linkedin folks was going to be on
> replication
> > > >    which will be a really disruptive set of features that need to be
> > > released
> > > >    together.
> > >
> > > I think there are some differing assumptions on the impact of
> > > replication.  My assumption has been that replication would be a new
> > > feature that was 'optional' in that R=1 would not be significantly from
> > > what we are doing now.  Thus replication work could continue in trunk
> --
> > > and even be present in releases -- without being disruptive, the lack
> > > change would just be to allow and document values of R > 1.
> > >
> > > However, both Jay and Jun's description of the work make it sound like
> > > more of a backwards incompatible no-rolling-restart every
> > > broker/consumer/producer needs to be upgraded at the same time sort of
> > > change.
> > >
> >
>

I agree this sounds reasonable.  In particular KAFKA-172, KAFKA-134, and
especially KAFKA-169 are examples of the kind of things I think are
worthwhile to get shipped into releases sooner rather than latter.

Thanks everyone.

On 11/14/2011 12:49 AM, Jay Kreps wrote:
> So the proposal I heard was:
>
>    - We create a branch for replication and begin development work on it
>    there.
>    - We leave trunk for incremental fixes and non-replication feature
>    development.
>    - We do monthly releases off trunk with whatever we have.
>    - We will number accordingly, so if we only have some small fixes then
>    the release will just be 0.7.1 or 0.7.01 or whatever; replication or larger
>    features will bump us up to 0.8.
>
> That seems reasonable to me. What this does mean, though is that LinkedIn
> may or may not end up running the releases in production before they go
> out, so this does put more testing/validation burden on the open source
> process.
>
> -Jay
>
> On Sun, Nov 13, 2011 at 6:47 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Hi, Chris,
>>
>> Yes, the changes related to replication are significant. It's actually a
>> bit hard to keep what we have now as the degenerated case with R=1.
>> Currently, the partions are numbered locally within each broker. This is
>> problematic with replication since a partition can have multiple replicas
>> that physically exist on multiple brokers. So, in our replication design,
>> partitions will be numbered globally within the cluster. This implies
>> potential on disk layout changes (see kafka-47) and wire protocol changes.
>>
>> You brought up a good question on migration path from 0.7 to 0.8. Since
>> replication changes are significant, it's going to be hard to make those
>> changes while keeping backward compatibility. My current feeling is that we
>> will make 0.8 a non-backward compatible release. To migrate to 0.8,
>> one possibility is to first create a new cluster of Kafka brokers using 0.8
>> (can potentially overlay on existing hardware). Then upgrade all producers
>> and point them to the new brokers. Drain all old consumers. Upgrade all
>> consumers and point them to the new brokers.
>>
>> What do people feel about this?
>>
>> Thanks,
>>
>> Jun
>>
>> On Sun, Nov 13, 2011 at 5:57 PM, Chris Burroughs
>> <[EMAIL PROTECTED]>wrote:
>>
>>> On 11/11/2011 06:01 PM, Jay Kreps wrote:
>>>> 2. Our current focus for linkedin folks was going to be on replication
>>>>    which will be a really disruptive set of features that need to be
>>> released
>>>>    together.
>>>
>>> I think there are some differing assumptions on the impact of
>>> replication.  My assumption has been that replication would be a new
>>> feature that was 'optional' in that R=1 would not be significantly from
>>> what we are doing now.  Thus replication work could continue in trunk --
>>> and even be present in releases -- without being disruptive, the lack
>>> change would just be to allow and document values of R > 1.
>>>
>>> However, both Jay and Jun's description of the work make it sound like
>>> more of a backwards incompatible no-rolling-restart every
>>> broker/consumer/producer needs to be upgraded at the same time sort of
>>> change.
>>>
>>
>