Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Possible check for speculative execution cancellation in finish() of storage UDF


Copy link to this message
-
Re: Possible check for speculative execution cancellation in finish() of storage UDF
Sandesh,

As a workaround you can set the property
mapred.[map|reduce].max.attempts to 1, which I believe will turn off
speculative execution. You can pass this as -D switch on pig command
line or through mapred-site.xml . Proper way to do it will be the way
you suggested (though that will be less performant as well as  complex
to implement). You may also want to comment on that jira with your
issue.

Ashutosh

On Tue, Apr 13, 2010 at 16:16, Sandesh Devaraju
<[EMAIL PROTECTED]> wrote:
> Hi All,
>
> I am using PIG-1229 to write pig query output to a database. However,
> I noticed that because of speculative execution, spurious records end
> up being written.
>
> I was wondering if there is a way to infer if current reduce task is
> running in a speculative slot that was cancelled (and hence a rollback
> needs to be issued).
>
> Thanks in advance!
>
> - Sandesh
>