Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Question about Avro "records"


+
Francis Galiegue 2013-02-27, 21:47
+
Pankaj Shroff 2013-02-27, 21:54
+
Francis Galiegue 2013-02-27, 22:14
+
Pankaj Shroff 2013-02-27, 22:21
+
Francis Galiegue 2013-02-27, 22:52
+
Doug Cutting 2013-02-27, 22:20
+
Francis Galiegue 2013-02-27, 22:51
+
Doug Cutting 2013-02-27, 23:10
+
Francis Galiegue 2013-02-27, 23:55
+
Doug Cutting 2013-02-28, 01:12
Copy link to this message
-
Re: Question about Avro "records"
There seems to be no way to easily use the avro libraries in Python (where
I feel most qualified to comment) to encode generics with "missing default
values" and have them transmitted in well-formed avro binary.

If you fill in the "missing" default values, the Python libraries will
transmit correctly.

I'd be happy to add methods to the avro.RecordSchema objects (in the Python
libraries) that "fill defaults" on missing member fields of a record,
recursively (which probably means method extension of other schema classes
as well). Shall I open a JIRA ticket for this for 1.7.5?

(Does providing this for Python pot me on the hook for such a thing in
other implementation languages? I hope not.)

For backwards compatibility (and probably to avoid unnecessary data
traversal), you'll probably want to explicitly ask the schema to fill in
defaults before transmission in the cases where you'd like to generate the
impoverished JSON from your example.

This seems related to earlier discussion today about designing constructors
to generate defaults already filled in.

Jeremy
On Feb 27, 2013 3:55 PM, "Francis Galiegue" <[EMAIL PROTECTED]> wrote:

> On Thu, Feb 28, 2013 at 12:10 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> > On Wed, Feb 27, 2013 at 2:51 PM, Francis Galiegue <[EMAIL PROTECTED]>
> wrote:
> >> Hmm, OK, the reader/writer distinction is something I am not accustomed
> to.
> >
> > http://avro.apache.org/docs/current/spec.html#Schema+Resolution
> >
> > Doug
>
> Hmmm, that does not quite answer my question about production of default
> values.
>
> For instance, when emitting data from an avro schema which reads:
>
> {
>     "type": "record",
>     "name": "whatever",
>     "fields": [ { "name": "a", "type": { "type": "int", "default": 0 } } ]
> }
>
> is emitting {} legal?
>
>
> --
> Francis Galiegue, [EMAIL PROTECTED]
> JSON Schema in Java: http://json-schema-validator.herokuapp.com
>
+
Doug Cutting 2013-02-28, 17:11
+
Francis Galiegue 2013-02-28, 01:40
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB