Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Is there a way to conditionally read Avro data?


Copy link to this message
-
RE: Is there a way to conditionally read Avro data?
Eric Wasserman 2013-08-16, 22:47
If you define you records like this (this is in the Avro IDL lang. for brevity)

If you write your records with a schema like this:
record R {

    Header header;

    Body body;

  }

Then you can read with a schema like this:
  record RSansBody {

    Header header;

  }

And the Avro libraries will read the header part (in which your "type" would reside) and effectively skip the body part.

________________________________
From: Anna Lahoud <[EMAIL PROTECTED]>
Sent: Friday, August 16, 2013 12:23 PM
To: [EMAIL PROTECTED]
Subject: Is there a way to conditionally read Avro data?

I am wondering if there is a way that I can avoid reading all of an item in an Avro file, based on some of the data that I have already read. For instance, say I have a datum where I know that if it's 'type' value is a 'ComputerVirus', and that I do not want to touch the remaining fields. Is there a way to 'move on' and get the next datum, without touching the remainder of the scary datum? I would call it a 'conditional read' in that I only want to fully read the datum if the datum meets some criteria.

Anna