Tatu Saloranta 2012-08-12, 22:26
Russell Jurney 2012-08-12, 22:34
Bill Graham 2012-08-13, 02:42
Russell Jurney 2012-08-13, 03:03
Tatu Saloranta 2012-08-13, 17:50
Bill Graham 2012-08-13, 22:59
Russell Jurney 2012-08-14, 00:31
1On Mon, Aug 13, 2012 at 3:59 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>> It is worth keeping in mind that explicit external schema is another
>> cost in not just designing but also maintaining the system. As such,
>> it is most useful for closely-coupled internal system, where one
>> controls both ends. This may be the case for computing pipelines a
>> single team owns.
> Our experiences have been quite the opposite. When the developer producing
> data was the same as the developer writing code to consume it, json worked
> fine since the developer knew what fields to expect. As our company grew,
> this turned into tribal knowledge and the approach did not scale. That's
> when having schemas is critical: when one team produces data and many others
> consume it. The cost is that the producer needs to publish the schema for
> others to discover.
Interesting, good point.
I was rather thinking of main cost being in maintenance, i.e. if and
when format changes, not so much upfront effort (although that's more
visible). And that cost depends on amount of change, if any, as well
as effort for other systems to adapt. Avro does have better support
for schema evolution, at least in theory. So that could help too.
-+ Tatu +-
Tatu Saloranta 2012-08-13, 17:47
Knoke, Jeff 2012-08-13, 12:49
Knoke, Jeff 2012-08-13, 12:46