Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Possible to include open .avro file in Map/Reduce job?


Copy link to this message
-
Possible to include open .avro file in Map/Reduce job?
Terry Healy 2013-01-14, 19:22
I have a log collection application that writes .avro files within HDFS.
Ideally I would like to include the current days (open for append) file
as one of the input files for a periodic M/R job.

I tried this but the Map job exited in error with the dreaded "Invalid
Sync!" IOException. I guess I should have expected this, but is there a
reasonable way around it? Can I catch the exception and just exit the
map at that point?

All suggestions appreciated.

-Terry