Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Is there a way to conditionally read Avro data?


Copy link to this message
-
RE: Is there a way to conditionally read Avro data?
If you define you records like this (this is in the Avro IDL lang. for brevity)

If you write your records with a schema like this:
record R {

    Header header;

    Body body;

  }

Then you can read with a schema like this:
  record RSansBody {

    Header header;

  }

And the Avro libraries will read the header part (in which your "type" would reside) and effectively skip the body part.

________________________________
From: Anna Lahoud <[EMAIL PROTECTED]>
Sent: Friday, August 16, 2013 12:23 PM
To: [EMAIL PROTECTED]
Subject: Is there a way to conditionally read Avro data?

I am wondering if there is a way that I can avoid reading all of an item in an Avro file, based on some of the data that I have already read. For instance, say I have a datum where I know that if it's 'type' value is a 'ComputerVirus', and that I do not want to touch the remaining fields. Is there a way to 'move on' and get the next datum, without touching the remainder of the scary datum? I would call it a 'conditional read' in that I only want to fully read the datum if the datum meets some criteria.

Anna

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB