Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Maintainer system?

Copy link to this message
Re: Maintainer system?
I think this idea has merits.  I know I am much more comfortable in coding
to the core kafka producer code and implementing kafka consumes but want to
spend more time in the log code and a few other places (time always a

I think we could get a lot more lean on things if we went with this type
of approach though I think the wiring protocol has to be something that
then bubbles up to either side producer/consumer ( on either fetch or

I think the ability and desire from the comunity to keep building
implementations of producer and consumer code also needs to keep a clean
and healthy growth
On Thu, Oct 4, 2012 at 4:13 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Hey guys,
> The number of developers and code base size for Kafka is getting larger.
> One way to help scale things gracefully would be to have an official idea
> of "subsystems" and have official maintainers for those. The duties of a
> maintainer would be
> 1. Be the final word on API design in that area
> 2. Ensure sufficient documentation and test coverage for that subsystem
> 3. Review all code changes in that sub-system area
> 4. Ensure that patches in that area get applied in a timely fashion
> In particular I think we could do a better job of getting patches in in a
> timely manner.
> Here are what I see as logically distinct systems or areas:
>    - Producer (java and scala)
>    - Consumer (java and scala)
>    - Network layer (kafka.network.*)
>    - Log (kafka.log.*)
>    - Replication (controller, fetcher threads, hw mark stuff, etc)
>    - Kafka API impl (basically just KafkaApi.scala)
>    - Hadoop stuff
>    - Perf tools and system tests
>    - Misc other small things: metrics, utils, etc.
> Obviously many features will cut across these layers, but the idea is that
> by having a real owner that is responsible for that area we will get higher
> quality.
> I think we are doing this informally already, but making it formal would
> help ensure you knew the right people to get input from. I think it
> probably wouldn't make sense to start doing this until post-0.8 since we
> are in the middle of so many things right now, but I wanted to see what
> people thought...?
> -Jay


Joe Stein
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>