Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # dev - Maintainer system?


+
Jay Kreps 2012-10-04, 20:13
Copy link to this message
-
Re: Maintainer system?
Joe Stein 2012-10-28, 05:06
I think this idea has merits.  I know I am much more comfortable in coding
to the core kafka producer code and implementing kafka consumes but want to
spend more time in the log code and a few other places (time always a
variable)

I think we could get a lot more lean on things if we went with this type
of approach though I think the wiring protocol has to be something that
then bubbles up to either side producer/consumer ( on either fetch or
response)

I think the ability and desire from the comunity to keep building
implementations of producer and consumer code also needs to keep a clean
and healthy growth
On Thu, Oct 4, 2012 at 4:13 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Hey guys,
>
> The number of developers and code base size for Kafka is getting larger.
>
> One way to help scale things gracefully would be to have an official idea
> of "subsystems" and have official maintainers for those. The duties of a
> maintainer would be
> 1. Be the final word on API design in that area
> 2. Ensure sufficient documentation and test coverage for that subsystem
> 3. Review all code changes in that sub-system area
> 4. Ensure that patches in that area get applied in a timely fashion
>
> In particular I think we could do a better job of getting patches in in a
> timely manner.
>
> Here are what I see as logically distinct systems or areas:
>
>    - Producer (java and scala)
>    - Consumer (java and scala)
>    - Network layer (kafka.network.*)
>    - Log (kafka.log.*)
>    - Replication (controller, fetcher threads, hw mark stuff, etc)
>    - Kafka API impl (basically just KafkaApi.scala)
>    - Hadoop stuff
>    - Perf tools and system tests
>    - Misc other small things: metrics, utils, etc.
>
> Obviously many features will cut across these layers, but the idea is that
> by having a real owner that is responsible for that area we will get higher
> quality.
>
> I think we are doing this informally already, but making it formal would
> help ensure you knew the right people to get input from. I think it
> probably wouldn't make sense to start doing this until post-0.8 since we
> are in the middle of so many things right now, but I wanted to see what
> people thought...?
>
> -Jay
>

--

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/
+
Jun Rao 2012-11-02, 05:15
+
Neha Narkhede 2012-11-02, 16:11
+
Prashanth Menon 2012-11-08, 22:40
+
Jay Kreps 2012-11-19, 18:48
+
Jay Kreps 2012-11-19, 19:08
+
Jakob Homan 2012-11-19, 19:24
+
Jay Kreps 2012-11-19, 19:30