Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Skipping Bad Records in M/R Job


Copy link to this message
-
Re: Skipping Bad Records in M/R Job
On Tue, Aug 9, 2011 at 5:28 PM, Maheshwaran Janarthanan <
[EMAIL PROTECTED]> wrote:

>
> Hi,
>
> I have written a Map reduce job which uses third party libraries to process
> unseen data which makes job fail because of errors in records.
>
> I realized 'Skipping Bad Records' feature in Hadoop Map/Reduce. Can Anyone
> send me the code snippet which enables this feature by setting properties on
> JobConf
>

I wouldn't recommend using the bad record skipping, since it was always
experimental and I don't think it has been well maintained.

If your 3rd part library crashes the jvm, I'd suggest using a subprocess to
call it and handle the errors yourself.

-- Owen