Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Skipping Bad Records in M/R Job

Copy link to this message
Re: Skipping Bad Records in M/R Job
On Tue, Aug 9, 2011 at 5:28 PM, Maheshwaran Janarthanan <

> Hi,
> I have written a Map reduce job which uses third party libraries to process
> unseen data which makes job fail because of errors in records.
> I realized 'Skipping Bad Records' feature in Hadoop Map/Reduce. Can Anyone
> send me the code snippet which enables this feature by setting properties on
> JobConf

I wouldn't recommend using the bad record skipping, since it was always
experimental and I don't think it has been well maintained.

If your 3rd part library crashes the jvm, I'd suggest using a subprocess to
call it and handle the errors yourself.

-- Owen