Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Possible to include open .avro file in Map/Reduce job?


Copy link to this message
-
Possible to include open .avro file in Map/Reduce job?
I have a log collection application that writes .avro files within HDFS.
Ideally I would like to include the current days (open for append) file
as one of the input files for a periodic M/R job.

I tried this but the Map job exited in error with the dreaded "Invalid
Sync!" IOException. I guess I should have expected this, but is there a
reasonable way around it? Can I catch the exception and just exit the
map at that point?

All suggestions appreciated.

-Terry
+
Doug Cutting 2013-01-17, 21:36
+
Terry Healy 2013-01-18, 14:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB