Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Config for new clients (and server)

Jay Kreps 2014-02-04, 17:34
David Arthur 2014-02-06, 14:28
Jay Kreps 2014-02-06, 16:46
Copy link to this message
Re: Config for new clients (and server)
Jay, thanks for the clarification. I think I understand it a bit better
now, and It all pretty much makes sense. See inline

Also, slightly OT, but what are Configurable and
AbstractConfig#getConfiguredInstance for? I don't see anything using them.

On 2/6/14 11:46 AM, Jay Kreps wrote:
> Hey David,
> You raise a couple points, let me give the rationale and see if you are
> convinced.
> 1. Why not reuse commons config? This is exactly the right question for
> application programming. We could save ourselves a few hundred lines of
> code we would maintain by just adding a library dependency, which is almost
> always the right decision if you are building an application as the
> application is itself the end result. But the issue here is that these
> clients will be embedded in thousands of applications which will in turn
> have other dependencies. So as a result any common libraries we use will
> also be used by others and this inevitably leads to
> dependency/incompatibility hell. As a result I really think for this kind
> of helper code we should just do it ourselves. I think of this as kind of
> our sacrifice for the users. :-)
> Meanwhile since we are decoupling the client and the server and since the
> server should always be a stand alone thing we should be able to be much
> more liberal in our dependencies there.
Yea, good point. I'd like to think that reusing existing stuff is always
the right thing to do, but I've been through dependency hell plenty to
know that's not the reality.
> 2. The next question is why not use a commons-config-like approach where
> the config object is a map/properties thing and we wrap it in a pojo that
> provides getters that parse values. This is the approach we took in the
> existing client and server code. The problem with this is that it is
> impossible to determine the set of configs programmatically. The goal was
> to have our config documentation automatically generated off the code
> (including the type, default value, documentation, etc). This also has the
> nice side effect that all the configs are validated up front, you don't
> have to wait until someone calls the getter to check validity.
I can see where the declarative stuff is useful for auto-generating
documentation, however that's another thing we'd have to build - that is
of course if you haven't built it already ;)

Loading the config eagerly also breaks the ability for config to change
at runtime, which can be appealing for changing things like buffer
sizes, timeouts, etc. I've been using Archaius lately, and it's pretty
awesome. But again, it's more geared towards application development
(like myself).
> 3. This approach doesn't prohibit custom validation. All validation is done
> with a Validator, so you can plug in any validation you like. This can be a
> bit ugly for one-off code (annonymous inner class, yuck...but will get
> better with java 8 and in scala is already good). But we can also just do
> validation in line in the wiring class. I see the "responsibility" of
> KafkaProducer or KafkaServer classes as turning configs into a constructed
> assembled hierachy of objects so checking stuff there is totally legit.
> 4. If the pojo wrapper you describe is just an internal convenience for our
> code (as Joel described), then I have no objection to that. But the
> objection to a user facing pojo was what I described before...
Internal/user-facing POJO might be nice so we _do_ get compiler warnings
when trying to get non-existent configs.

> Make sense?
> -Jay
> On Thu, Feb 6, 2014 at 6:27 AM, David Arthur <[EMAIL PROTECTED]> wrote:
>> The declarative approach strikes me as a bit odd. Why not put the
>> precondition logic in a POJO wrapper? Also, why reinvent
>> commons-configuration? It's got a nice API and tons of people use it.
>> public class ProducerConfig extends org.apache.commons.configuration.
>> AbstractConfiguration {
>>    /**
>>      * Blah, short doc @{see #getRequiredAcks}
Jay Kreps 2014-02-06, 21:32
Jun Rao 2014-02-10, 05:24
Jay Kreps 2014-02-10, 18:16
Jun Rao 2014-02-10, 22:02
Jay Kreps 2014-02-11, 00:13
Jun Rao 2014-02-12, 05:30
Steve Morin 2014-02-12, 05:36
Guozhang Wang 2014-02-06, 01:31
Jay Kreps 2014-02-06, 16:30
Guozhang Wang 2014-02-06, 17:52
Jay Kreps 2014-02-06, 18:54
Jay Kreps 2014-02-06, 16:33
Pradeep Gollakota 2014-02-10, 23:10