Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Sync() between records? How do we recover from a bad record, using DataFileReader?


+
Russell Jurney 2013-01-07, 03:38
Copy link to this message
-
Re: Sync() between records? How do we recover from a bad record, using DataFileReader?
For the corruption test, try corrupting the records, not the sync marker.
The features added to DataFileReader for corruption recovery were for the
case when decoding a record fails (corrupted record), not for when a sync
marker is corrupted.  Perhaps we should add that too, but it does not
surprise me that that case has a bug.
On 1/6/13 7:38 PM, "Russell Jurney" <[EMAIL PROTECTED]> wrote:
>We are trying to recover, report bad record, and move to the next record
>of an Avro file in PIG-3015 and PIG-3059. It seems that sync blocks don't
>exist between files, however.
>
>How should we recover from a bad record using Avro's DataFileReader?
>
>https://issues.apache.org/jira/browse/PIG-3015
>https://issues.apache.org/jira/browse/PIG-3059
>
>Russell Jurney http://datasyndrome.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB