Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Changing Avro schemas for daily imports


Copy link to this message
-
Changing Avro schemas for daily imports
Hello!

Every night, we fetch mysql rows with a timestamp of the day before,
and store them into avro, creating a Y-M-d.avro file on HDFS daily.

This is the schema:

{
   "namespace": "asp",
   "type": "record",
   "name": "trace",
   "fields": [
     {
       "type": "long",
       "name": "id"
     },
     {
       "type": "long",
       "name": "timestamp"
     },
     {
       "type": [
         "int",
         "null"
       ],
       "name": "latitude"
     },
     {
       "type": [
         "int",
         "null"
       ],
       "name": "longitude"
     }
   ]
}

Now i would like to change timestamp so it can be null as well. The
plan is to just change the timestamp type in the schema starting from
the next day. I'm pretty sure it won't affect lookups in any way ( like
when using Pig ), but I thought I would ask to be sure ( since the
structure itself doesn't change, only the type ). I wouldn't want to run
into a gotcha after months of importing with my adjusted schema. :)

Thanks in advance!

Kind regards,

Bart
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB