-Possible to include open .avro file in Map/Reduce job?
Terry Healy 2013-01-14, 19:22
I have a log collection application that writes .avro files within HDFS.
Ideally I would like to include the current days (open for append) file
as one of the input files for a periodic M/R job.
I tried this but the Map job exited in error with the dreaded "Invalid
Sync!" IOException. I guess I should have expected this, but is there a
reasonable way around it? Can I catch the exception and just exit the
map at that point?
All suggestions appreciated.
Doug Cutting 2013-01-17, 21:36
Terry Healy 2013-01-18, 14:51