Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Adding extra fields into the schema


+
Harsh J 2010-07-17, 17:45
+
Harsh J 2010-07-19, 06:38
+
Doug Cutting 2010-07-19, 18:40
Copy link to this message
-
Re: Adding extra fields into the schema
Its pretty odd, the test case passes (Tried trunk though) but I'm
unable to add properties. Was this fixed for 1.4 or 1.3 also? I'm
using 1.3.3 as of now.

All I do is create a schema (with properties added to the schema
object for the chosen fields) at the InputFormat level (pre-Mapper),
place it in the JobConf with a toString and get it out again within my
Mapper and create a Schema from it. I lose my set properties, but the
rest remains perfectly intact. I'm only adding properties to non-union
fields of my record.

On Tue, Jul 20, 2010 at 12:10 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> This should work.
>
> There are existing tests for schema properties are in TestSchema.java. The
> checkProp() method is called to add a property to every type of schema,
> print the schema with that property, then re-parse it and check that the
> property is still there.
>
> Can you provide a complete test case that illustrates this failing?
>
> Thanks,
>
> Doug
>
> On 07/17/2010 10:45 AM, Harsh J wrote:
>>
>> I'm wondering if there's a way to add user-defined name:value pairs
>> into the Avro JSON schema itself?
>>
>> For instance, while building my schema (by using setFields for a
>> collection of fields), for some fields I do the following: [Using Java
>> API]
>>
>> fieldObject.schema().addProp("myproperty", "myvalue");
>>
>> I see that it adds to the schema like this:
>> {
>>     "name" : "someFieldXYZ",
>>     "type" : {
>>       "type" : "bytes",
>>       "myproperty" : "myvalue"
>>     },
>>     "default" : "XYZ"
>> }
>>
>> But when I re-parse this created schema and try to retrieve my
>> property back, I get a null. I do this for getting it back:
>>
>> fieldObject.schema.getProp("myproperty");
>>
>> Why am I getting a null here? How do I access what's I embedded into
>> the schema properly back again? Or is this plain WRONG a thing to do?
>>
>> [Type resolution still works fine even after it getting modified like
>> that].
>>
>

--
Harsh J
www.harshj.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB