Hey guys,

The number of developers and code base size for Kafka is getting larger.

One way to help scale things gracefully would be to have an official idea
of "subsystems" and have official maintainers for those. The duties of a
maintainer would be
1. Be the final word on API design in that area
2. Ensure sufficient documentation and test coverage for that subsystem
3. Review all code changes in that sub-system area
4. Ensure that patches in that area get applied in a timely fashion

In particular I think we could do a better job of getting patches in in a
timely manner.

Here are what I see as logically distinct systems or areas:

   - Producer (java and scala)
   - Consumer (java and scala)
   - Network layer (kafka.network.*)
   - Log (kafka.log.*)
   - Replication (controller, fetcher threads, hw mark stuff, etc)
   - Kafka API impl (basically just KafkaApi.scala)
   - Hadoop stuff
   - Perf tools and system tests
   - Misc other small things: metrics, utils, etc.

Obviously many features will cut across these layers, but the idea is that
by having a real owner that is responsible for that area we will get higher
quality.

I think we are doing this informally already, but making it formal would
help ensure you knew the right people to get input from. I think it
probably wouldn't make sense to start doing this until post-0.8 since we
are in the middle of so many things right now, but I wanted to see what
people thought...?

-Jay

I think this idea has merits.  I know I am much more comfortable in coding
to the core kafka producer code and implementing kafka consumes but want to
spend more time in the log code and a few other places (time always a
variable)

I think we could get a lot more lean on things if we went with this type
of approach though I think the wiring protocol has to be something that
then bubbles up to either side producer/consumer ( on either fetch or
response)

I think the ability and desire from the comunity to keep building
implementations of producer and consumer code also needs to keep a clean
and healthy growth
On Thu, Oct 4, 2012 at 4:13 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Hey guys,
>
> The number of developers and code base size for Kafka is getting larger.
>
> One way to help scale things gracefully would be to have an official idea
> of "subsystems" and have official maintainers for those. The duties of a
> maintainer would be
> 1. Be the final word on API design in that area
> 2. Ensure sufficient documentation and test coverage for that subsystem
> 3. Review all code changes in that sub-system area
> 4. Ensure that patches in that area get applied in a timely fashion
>
> In particular I think we could do a better job of getting patches in in a
> timely manner.
>
> Here are what I see as logically distinct systems or areas:
>
>    - Producer (java and scala)
>    - Consumer (java and scala)
>    - Network layer (kafka.network.*)
>    - Log (kafka.log.*)
>    - Replication (controller, fetcher threads, hw mark stuff, etc)
>    - Kafka API impl (basically just KafkaApi.scala)
>    - Hadoop stuff
>    - Perf tools and system tests
>    - Misc other small things: metrics, utils, etc.
>
> Obviously many features will cut across these layers, but the idea is that
> by having a real owner that is responsible for that area we will get higher
> quality.
>
> I think we are doing this informally already, but making it formal would
> help ensure you knew the right people to get input from. I think it
> probably wouldn't make sense to start doing this until post-0.8 since we
> are in the middle of so many things right now, but I wanted to see what
> people thought...?
>
> -Jay
>

--

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/

I think that's a good idea. It will be good to have at least 2 maintainers
per component.

I'd encourage more people to review patches. The more patches one reviews,
the more familiar he/she is with the components.

Thanks,

Jun

On Thu, Oct 4, 2012 at 1:13 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Hey guys,
>
> The number of developers and code base size for Kafka is getting larger.
>
> One way to help scale things gracefully would be to have an official idea
> of "subsystems" and have official maintainers for those. The duties of a
> maintainer would be
> 1. Be the final word on API design in that area
> 2. Ensure sufficient documentation and test coverage for that subsystem
> 3. Review all code changes in that sub-system area
> 4. Ensure that patches in that area get applied in a timely fashion
>
> In particular I think we could do a better job of getting patches in in a
> timely manner.
>
> Here are what I see as logically distinct systems or areas:
>
>    - Producer (java and scala)
>    - Consumer (java and scala)
>    - Network layer (kafka.network.*)
>    - Log (kafka.log.*)
>    - Replication (controller, fetcher threads, hw mark stuff, etc)
>    - Kafka API impl (basically just KafkaApi.scala)
>    - Hadoop stuff
>    - Perf tools and system tests
>    - Misc other small things: metrics, utils, etc.
>
> Obviously many features will cut across these layers, but the idea is that
> by having a real owner that is responsible for that area we will get higher
> quality.
>
> I think we are doing this informally already, but making it formal would
> help ensure you knew the right people to get input from. I think it
> probably wouldn't make sense to start doing this until post-0.8 since we
> are in the middle of so many things right now, but I wanted to see what
> people thought...?
>
> -Jay
>

+1 on this, I like the idea of having a primary/secondary owner system
for each component.

Thanks,
Neha

On Thu, Nov 1, 2012 at 10:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> I think that's a good idea. It will be good to have at least 2 maintainers
> per component.
>
> I'd encourage more people to review patches. The more patches one reviews,
> the more familiar he/she is with the components.
>
> Thanks,
>
> Jun
>
> On Thu, Oct 4, 2012 at 1:13 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> Hey guys,
>>
>> The number of developers and code base size for Kafka is getting larger.
>>
>> One way to help scale things gracefully would be to have an official idea
>> of "subsystems" and have official maintainers for those. The duties of a
>> maintainer would be
>> 1. Be the final word on API design in that area
>> 2. Ensure sufficient documentation and test coverage for that subsystem
>> 3. Review all code changes in that sub-system area
>> 4. Ensure that patches in that area get applied in a timely fashion
>>
>> In particular I think we could do a better job of getting patches in in a
>> timely manner.
>>
>> Here are what I see as logically distinct systems or areas:
>>
>>    - Producer (java and scala)
>>    - Consumer (java and scala)
>>    - Network layer (kafka.network.*)
>>    - Log (kafka.log.*)
>>    - Replication (controller, fetcher threads, hw mark stuff, etc)
>>    - Kafka API impl (basically just KafkaApi.scala)
>>    - Hadoop stuff
>>    - Perf tools and system tests
>>    - Misc other small things: metrics, utils, etc.
>>
>> Obviously many features will cut across these layers, but the idea is that
>> by having a real owner that is responsible for that area we will get higher
>> quality.
>>
>> I think we are doing this informally already, but making it formal would
>> help ensure you knew the right people to get input from. I think it
>> probably wouldn't make sense to start doing this until post-0.8 since we
>> are in the middle of so many things right now, but I wanted to see what
>> people thought...?
>>
>> -Jay
>>

+1 from me on the maintainer system and having a primary/secondary for
components.

I also think we should try as much as possible to get at least two
reviewers for patches that come in.  This is something I'm very guilty of
and am trying to correct.  I get the feeling Jun is overwhelmed with patch
reviews :)

- Prashanth

On Fri, Nov 2, 2012 at 12:11 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> +1 on this, I like the idea of having a primary/secondary owner system
> for each component.
>
> Thanks,
> Neha
>
> On Thu, Nov 1, 2012 at 10:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > I think that's a good idea. It will be good to have at least 2
> maintainers
> > per component.
> >
> > I'd encourage more people to review patches. The more patches one
> reviews,
> > the more familiar he/she is with the components.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Oct 4, 2012 at 1:13 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> Hey guys,
> >>
> >> The number of developers and code base size for Kafka is getting larger.
> >>
> >> One way to help scale things gracefully would be to have an official
> idea
> >> of "subsystems" and have official maintainers for those. The duties of a
> >> maintainer would be
> >> 1. Be the final word on API design in that area
> >> 2. Ensure sufficient documentation and test coverage for that subsystem
> >> 3. Review all code changes in that sub-system area
> >> 4. Ensure that patches in that area get applied in a timely fashion
> >>
> >> In particular I think we could do a better job of getting patches in in
> a
> >> timely manner.
> >>
> >> Here are what I see as logically distinct systems or areas:
> >>
> >>    - Producer (java and scala)
> >>    - Consumer (java and scala)
> >>    - Network layer (kafka.network.*)
> >>    - Log (kafka.log.*)
> >>    - Replication (controller, fetcher threads, hw mark stuff, etc)
> >>    - Kafka API impl (basically just KafkaApi.scala)
> >>    - Hadoop stuff
> >>    - Perf tools and system tests
> >>    - Misc other small things: metrics, utils, etc.
> >>
> >> Obviously many features will cut across these layers, but the idea is
> that
> >> by having a real owner that is responsible for that area we will get
> higher
> >> quality.
> >>
> >> I think we are doing this informally already, but making it formal would
> >> help ensure you knew the right people to get input from. I think it
> >> probably wouldn't make sense to start doing this until post-0.8 since we
> >> are in the middle of so many things right now, but I wanted to see what
> >> people thought...?
> >>
> >> -Jay
> >>
>

Jun is just far and away the best reviewer. So maybe we should just require
that everybody get Jun to review their patches. :-)

Or, more practically, maybe Jun should put together some guidelines on what
he does and we can try to emulate.

-Jay

On Thu, Nov 8, 2012 at 2:40 PM, Prashanth Menon
<[EMAIL PROTECTED]>wrote:

> +1 from me on the maintainer system and having a primary/secondary for
> components.
>
> I also think we should try as much as possible to get at least two
> reviewers for patches that come in.  This is something I'm very guilty of
> and am trying to correct.  I get the feeling Jun is overwhelmed with patch
> reviews :)
>
> - Prashanth
>
> On Fri, Nov 2, 2012 at 12:11 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > +1 on this, I like the idea of having a primary/secondary owner system
> > for each component.
> >
> > Thanks,
> > Neha
> >
> > On Thu, Nov 1, 2012 at 10:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > I think that's a good idea. It will be good to have at least 2
> > maintainers
> > > per component.
> > >
> > > I'd encourage more people to review patches. The more patches one
> > reviews,
> > > the more familiar he/she is with the components.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Oct 4, 2012 at 1:13 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> > >
> > >> Hey guys,
> > >>
> > >> The number of developers and code base size for Kafka is getting
> larger.
> > >>
> > >> One way to help scale things gracefully would be to have an official
> > idea
> > >> of "subsystems" and have official maintainers for those. The duties
> of a
> > >> maintainer would be
> > >> 1. Be the final word on API design in that area
> > >> 2. Ensure sufficient documentation and test coverage for that
> subsystem
> > >> 3. Review all code changes in that sub-system area
> > >> 4. Ensure that patches in that area get applied in a timely fashion
> > >>
> > >> In particular I think we could do a better job of getting patches in
> in
> > a
> > >> timely manner.
> > >>
> > >> Here are what I see as logically distinct systems or areas:
> > >>
> > >>    - Producer (java and scala)
> > >>    - Consumer (java and scala)
> > >>    - Network layer (kafka.network.*)
> > >>    - Log (kafka.log.*)
> > >>    - Replication (controller, fetcher threads, hw mark stuff, etc)
> > >>    - Kafka API impl (basically just KafkaApi.scala)
> > >>    - Hadoop stuff
> > >>    - Perf tools and system tests
> > >>    - Misc other small things: metrics, utils, etc.
> > >>
> > >> Obviously many features will cut across these layers, but the idea is
> > that
> > >> by having a real owner that is responsible for that area we will get
> > higher
> > >> quality.
> > >>
> > >> I think we are doing this informally already, but making it formal
> would
> > >> help ensure you knew the right people to get input from. I think it
> > >> probably wouldn't make sense to start doing this until post-0.8 since
> we
> > >> are in the middle of so many things right now, but I wanted to see
> what
> > >> people thought...?
> > >>
> > >> -Jay
> > >>
> >
>

Okay so there seems to be a consensus that this is a good idea. I have
added a wiki page with maintainers:
  https://cwiki.apache.org/confluence/display/KAFKA/Maintainers
Could committers do the following:
1. Add themselves to areas they would like to act as a maintainer for? Our
goal will be two have two maintainers for each area, but if we have either
one or three that is fine too. Sign up for areas where you think you have
expertise or can make a difference.
2. Add other areas I left out. I think it is fine if they are small. For
example I think it would be fine to have a single maintainer for
Utils.scala even though that is not a major area.

-Jay
On Mon, Nov 19, 2012 at 10:48 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Jun is just far and away the best reviewer. So maybe we should just
> require that everybody get Jun to review their patches. :-)
>
> Or, more practically, maybe Jun should put together some guidelines on
> what he does and we can try to emulate.
>
> -Jay
>
>
> On Thu, Nov 8, 2012 at 2:40 PM, Prashanth Menon <
> [EMAIL PROTECTED]> wrote:
>
>> +1 from me on the maintainer system and having a primary/secondary for
>> components.
>>
>> I also think we should try as much as possible to get at least two
>> reviewers for patches that come in.  This is something I'm very guilty of
>> and am trying to correct.  I get the feeling Jun is overwhelmed with patch
>> reviews :)
>>
>> - Prashanth
>>
>> On Fri, Nov 2, 2012 at 12:11 PM, Neha Narkhede <[EMAIL PROTECTED]
>> >wrote:
>>
>> > +1 on this, I like the idea of having a primary/secondary owner system
>> > for each component.
>> >
>> > Thanks,
>> > Neha
>> >
>> > On Thu, Nov 1, 2012 at 10:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > > I think that's a good idea. It will be good to have at least 2
>> > maintainers
>> > > per component.
>> > >
>> > > I'd encourage more people to review patches. The more patches one
>> > reviews,
>> > > the more familiar he/she is with the components.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Thu, Oct 4, 2012 at 1:13 PM, Jay Kreps <[EMAIL PROTECTED]>
>> wrote:
>> > >
>> > >> Hey guys,
>> > >>
>> > >> The number of developers and code base size for Kafka is getting
>> larger.
>> > >>
>> > >> One way to help scale things gracefully would be to have an official
>> > idea
>> > >> of "subsystems" and have official maintainers for those. The duties
>> of a
>> > >> maintainer would be
>> > >> 1. Be the final word on API design in that area
>> > >> 2. Ensure sufficient documentation and test coverage for that
>> subsystem
>> > >> 3. Review all code changes in that sub-system area
>> > >> 4. Ensure that patches in that area get applied in a timely fashion
>> > >>
>> > >> In particular I think we could do a better job of getting patches in
>> in
>> > a
>> > >> timely manner.
>> > >>
>> > >> Here are what I see as logically distinct systems or areas:
>> > >>
>> > >>    - Producer (java and scala)
>> > >>    - Consumer (java and scala)
>> > >>    - Network layer (kafka.network.*)
>> > >>    - Log (kafka.log.*)
>> > >>    - Replication (controller, fetcher threads, hw mark stuff, etc)
>> > >>    - Kafka API impl (basically just KafkaApi.scala)
>> > >>    - Hadoop stuff
>> > >>    - Perf tools and system tests
>> > >>    - Misc other small things: metrics, utils, etc.
>> > >>
>> > >> Obviously many features will cut across these layers, but the idea is
>> > that
>> > >> by having a real owner that is responsible for that area we will get
>> > higher
>> > >> quality.
>> > >>
>> > >> I think we are doing this informally already, but making it formal
>> would
>> > >> help ensure you knew the right people to get input from. I think it
>> > >> probably wouldn't make sense to start doing this until post-0.8
>> since we
>> > >> are in the middle of so many things right now, but I wanted to see
>> what
>> > >> people thought...?
>> > >>
>> > >> -Jay
>> > >>
>> >
>>
>
>

One thing to keep in mind is that Apache dissuades against the @author
tag to try and ensure the whole community, particularly the
committers, feels ownership of the whole codebase.  While committers
not familiar with a particular bit of code shouldn't necessarily
review a piece of code, neither should any one subset of the committer
community be the only ones able to move a section of the codebase
forward.

Basically, we should just make sure that this system is not used to
fence people in or restrict proven contributors' ability to work in
one section or another.

On 19 November 2012 11:08, Jay Kreps <[EMAIL PROTECTED]> wrote:
> Okay so there seems to be a consensus that this is a good idea. I have
> added a wiki page with maintainers:
>   https://cwiki.apache.org/confluence/display/KAFKA/Maintainers
> Could committers do the following:
> 1. Add themselves to areas they would like to act as a maintainer for? Our
> goal will be two have two maintainers for each area, but if we have either
> one or three that is fine too. Sign up for areas where you think you have
> expertise or can make a difference.
> 2. Add other areas I left out. I think it is fine if they are small. For
> example I think it would be fine to have a single maintainer for
> Utils.scala even though that is not a major area.
>
> -Jay
>
>
> On Mon, Nov 19, 2012 at 10:48 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> Jun is just far and away the best reviewer. So maybe we should just
>> require that everybody get Jun to review their patches. :-)
>>
>> Or, more practically, maybe Jun should put together some guidelines on
>> what he does and we can try to emulate.
>>
>> -Jay
>>
>>
>> On Thu, Nov 8, 2012 at 2:40 PM, Prashanth Menon <
>> [EMAIL PROTECTED]> wrote:
>>
>>> +1 from me on the maintainer system and having a primary/secondary for
>>> components.
>>>
>>> I also think we should try as much as possible to get at least two
>>> reviewers for patches that come in.  This is something I'm very guilty of
>>> and am trying to correct.  I get the feeling Jun is overwhelmed with patch
>>> reviews :)
>>>
>>> - Prashanth
>>>
>>> On Fri, Nov 2, 2012 at 12:11 PM, Neha Narkhede <[EMAIL PROTECTED]
>>> >wrote:
>>>
>>> > +1 on this, I like the idea of having a primary/secondary owner system
>>> > for each component.
>>> >
>>> > Thanks,
>>> > Neha
>>> >
>>> > On Thu, Nov 1, 2012 at 10:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>> > > I think that's a good idea. It will be good to have at least 2
>>> > maintainers
>>> > > per component.
>>> > >
>>> > > I'd encourage more people to review patches. The more patches one
>>> > reviews,
>>> > > the more familiar he/she is with the components.
>>> > >
>>> > > Thanks,
>>> > >
>>> > > Jun
>>> > >
>>> > > On Thu, Oct 4, 2012 at 1:13 PM, Jay Kreps <[EMAIL PROTECTED]>
>>> wrote:
>>> > >
>>> > >> Hey guys,
>>> > >>
>>> > >> The number of developers and code base size for Kafka is getting
>>> larger.
>>> > >>
>>> > >> One way to help scale things gracefully would be to have an official
>>> > idea
>>> > >> of "subsystems" and have official maintainers for those. The duties
>>> of a
>>> > >> maintainer would be
>>> > >> 1. Be the final word on API design in that area
>>> > >> 2. Ensure sufficient documentation and test coverage for that
>>> subsystem
>>> > >> 3. Review all code changes in that sub-system area
>>> > >> 4. Ensure that patches in that area get applied in a timely fashion
>>> > >>
>>> > >> In particular I think we could do a better job of getting patches in
>>> in
>>> > a
>>> > >> timely manner.
>>> > >>
>>> > >> Here are what I see as logically distinct systems or areas:
>>> > >>
>>> > >>    - Producer (java and scala)
>>> > >>    - Consumer (java and scala)
>>> > >>    - Network layer (kafka.network.*)
>>> > >>    - Log (kafka.log.*)
>>> > >>    - Replication (controller, fetcher threads, hw mark stuff, etc)
>>> > >>    - Kafka API impl (basically just KafkaApi.scala)
>>> > >>    - Hadoop stuff
>>> > >>    - Perf tools and system tests

Agreed. It is clearly a balancing act. Let us know if you think we are off
the reservation.

The two practical problems I want to solve:
1. We are too slow at reviewing patches, this makes it so that each patch
has at least one other person who knows they need to look at it asap.
2. Cross cutting changes seem to lead to hacks in some areas. For example
the producer code definitely got nastier during the replication work. I
feel the cause is that when working on a feature you sometimes need to work
on code you don't know that well, and consequently you don't really know
the right thing to do. I am certainly guilty of this. Having someone who is
specifically responsible for that will lead to greater incentive to catch
this in review and increase the change that the right person will be doing
the review.

-Jay
On Mon, Nov 19, 2012 at 11:24 AM, Jakob Homan <[EMAIL PROTECTED]> wrote:

> One thing to keep in mind is that Apache dissuades against the @author
> tag to try and ensure the whole community, particularly the
> committers, feels ownership of the whole codebase.  While committers
> not familiar with a particular bit of code shouldn't necessarily
> review a piece of code, neither should any one subset of the committer
> community be the only ones able to move a section of the codebase
> forward.
>
> Basically, we should just make sure that this system is not used to
> fence people in or restrict proven contributors' ability to work in
> one section or another.
>
> On 19 November 2012 11:08, Jay Kreps <[EMAIL PROTECTED]> wrote:
> > Okay so there seems to be a consensus that this is a good idea. I have
> > added a wiki page with maintainers:
> >   https://cwiki.apache.org/confluence/display/KAFKA/Maintainers
> > Could committers do the following:
> > 1. Add themselves to areas they would like to act as a maintainer for?
> Our
> > goal will be two have two maintainers for each area, but if we have
> either
> > one or three that is fine too. Sign up for areas where you think you have
> > expertise or can make a difference.
> > 2. Add other areas I left out. I think it is fine if they are small. For
> > example I think it would be fine to have a single maintainer for
> > Utils.scala even though that is not a major area.
> >
> > -Jay
> >
> >
> > On Mon, Nov 19, 2012 at 10:48 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> Jun is just far and away the best reviewer. So maybe we should just
> >> require that everybody get Jun to review their patches. :-)
> >>
> >> Or, more practically, maybe Jun should put together some guidelines on
> >> what he does and we can try to emulate.
> >>
> >> -Jay
> >>
> >>
> >> On Thu, Nov 8, 2012 at 2:40 PM, Prashanth Menon <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >>> +1 from me on the maintainer system and having a primary/secondary for
> >>> components.
> >>>
> >>> I also think we should try as much as possible to get at least two
> >>> reviewers for patches that come in.  This is something I'm very guilty
> of
> >>> and am trying to correct.  I get the feeling Jun is overwhelmed with
> patch
> >>> reviews :)
> >>>
> >>> - Prashanth
> >>>
> >>> On Fri, Nov 2, 2012 at 12:11 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> >>> >wrote:
> >>>
> >>> > +1 on this, I like the idea of having a primary/secondary owner
> system
> >>> > for each component.
> >>> >
> >>> > Thanks,
> >>> > Neha
> >>> >
> >>> > On Thu, Nov 1, 2012 at 10:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >>> > > I think that's a good idea. It will be good to have at least 2
> >>> > maintainers
> >>> > > per component.
> >>> > >
> >>> > > I'd encourage more people to review patches. The more patches one
> >>> > reviews,
> >>> > > the more familiar he/she is with the components.
> >>> > >
> >>> > > Thanks,
> >>> > >
> >>> > > Jun
> >>> > >
> >>> > > On Thu, Oct 4, 2012 at 1:13 PM, Jay Kreps <[EMAIL PROTECTED]>
> >>> wrote:
> >>> > >
> >>> > >> Hey guys,
> >>> > >>
> >>> > >> The number of developers and code base size for Kafka is getting