+
Russell Jurney 2012-03-02, 00:19
+
Bill Graham 2012-03-02, 05:37
+
Dmitriy Ryaboy 2012-03-02, 21:45
+
Bill Graham 2012-03-02, 21:49
+
Russell Jurney 2012-03-02, 22:40
+
Bill Graham 2012-03-02, 23:18
+
Corbin Hoenes 2013-01-11, 17:14
-
Re: Turn off Speculative Execution in a UDF?Alan Gates 2013-01-11, 17:18
Store functions can run in either map or reduce depending on your script. If your script has any operator that requires a reduce (most joins, group by, order by, distinct, limit) then the store function will be in a reduce.
Alan.
On Jan 11, 2013, at 9:14 AM, Corbin Hoenes wrote:
> Hi all,
>
> I am a little unclear about which speculative execution you must disable.
> What phase do storage functions run at map or reduce?
>
> I've always just done both.
> set mapred.reduce.tasks.speculative.execution false
> set mapred.map.tasks.speculative.execution false
>
> Thanks for any hints!
>
> On Fri, Mar 2, 2012 at 4:18 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> I tried to test turning this off in the setStoreLocation method but without
>> that change I wasn't able to get a job to run with SE happening. As a
>> result I can't verify that the setting is doing anything. Russell, if you
>> can reproduce SE I'd be curious to hear if you could turn it off in
>> setStoreLocation.
>>
>> On Fri, Mar 2, 2012 at 2:40 PM, Russell Jurney <[EMAIL PROTECTED]
>>> wrote:
>>
>>> I thought it was too late in the workflow to do this, but it would be
>>> really cool if you could. I don't think to think about MapReduce much
>> when
>>> I Pig, except to group my scripts by jobs... so this was a surprise for
>> me.
>>> Made sense once i thought of it. But it was a surprise.
>>>
>>>
>>> On Fri, Mar 2, 2012 at 1:49 PM, Bill Graham <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> I was also curious about this and will try it, but my initial thought
>> was
>>>> that at that point it might be tool late in the workflow of the job.
>> I'll
>>>> give it a shot and report back.
>>>>
>>>>
>>>> On Fri, Mar 2, 2012 at 1:45 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>>> In a StoreFunc, you could do that when you get passed the jobconf,
>>>> right?
>>>>>
>>>>> On Thu, Mar 1, 2012 at 9:37 PM, Bill Graham <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>> I don't think so. We just do it in the pig script before using the
>>>> store
>>>>>> func:
>>>>>>
>>>>>> SET mapred.map.tasks.speculative.execution false
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 1, 2012 at 4:19 PM, Russell Jurney <
>>>> [EMAIL PROTECTED]
>>>>>> wrote:
>>>>>>
>>>>>>> Can you turn off speculative execution in a StoreFunc? I believe
>> it
>>>> is
>>>>>>> leading to duplicates in MongoStorage();
>>>>>>>
>>>>>>> --
>>>>>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
>>>>>>> datasyndrome.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Note that I'm no longer using my Yahoo! email address. Please email
>>>> me
>>>>> at
>>>>>> [EMAIL PROTECTED] going forward.*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Note that I'm no longer using my Yahoo! email address. Please email me
>> at
>>>> [EMAIL PROTECTED] going forward.*
>>>>
>>>
>>>
>>>
>>> --
>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]atasyndrome.
>>> com
>>>
>>
>>
>>
>> --
>> *Note that I'm no longer using my Yahoo! email address. Please email me at
>> [EMAIL PROTECTED] going forward.*
>>
Alan.
On Jan 11, 2013, at 9:14 AM, Corbin Hoenes wrote:
> Hi all,
>
> I am a little unclear about which speculative execution you must disable.
> What phase do storage functions run at map or reduce?
>
> I've always just done both.
> set mapred.reduce.tasks.speculative.execution false
> set mapred.map.tasks.speculative.execution false
>
> Thanks for any hints!
>
> On Fri, Mar 2, 2012 at 4:18 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> I tried to test turning this off in the setStoreLocation method but without
>> that change I wasn't able to get a job to run with SE happening. As a
>> result I can't verify that the setting is doing anything. Russell, if you
>> can reproduce SE I'd be curious to hear if you could turn it off in
>> setStoreLocation.
>>
>> On Fri, Mar 2, 2012 at 2:40 PM, Russell Jurney <[EMAIL PROTECTED]
>>> wrote:
>>
>>> I thought it was too late in the workflow to do this, but it would be
>>> really cool if you could. I don't think to think about MapReduce much
>> when
>>> I Pig, except to group my scripts by jobs... so this was a surprise for
>> me.
>>> Made sense once i thought of it. But it was a surprise.
>>>
>>>
>>> On Fri, Mar 2, 2012 at 1:49 PM, Bill Graham <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> I was also curious about this and will try it, but my initial thought
>> was
>>>> that at that point it might be tool late in the workflow of the job.
>> I'll
>>>> give it a shot and report back.
>>>>
>>>>
>>>> On Fri, Mar 2, 2012 at 1:45 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>>> In a StoreFunc, you could do that when you get passed the jobconf,
>>>> right?
>>>>>
>>>>> On Thu, Mar 1, 2012 at 9:37 PM, Bill Graham <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>> I don't think so. We just do it in the pig script before using the
>>>> store
>>>>>> func:
>>>>>>
>>>>>> SET mapred.map.tasks.speculative.execution false
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 1, 2012 at 4:19 PM, Russell Jurney <
>>>> [EMAIL PROTECTED]
>>>>>> wrote:
>>>>>>
>>>>>>> Can you turn off speculative execution in a StoreFunc? I believe
>> it
>>>> is
>>>>>>> leading to duplicates in MongoStorage();
>>>>>>>
>>>>>>> --
>>>>>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
>>>>>>> datasyndrome.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Note that I'm no longer using my Yahoo! email address. Please email
>>>> me
>>>>> at
>>>>>> [EMAIL PROTECTED] going forward.*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Note that I'm no longer using my Yahoo! email address. Please email me
>> at
>>>> [EMAIL PROTECTED] going forward.*
>>>>
>>>
>>>
>>>
>>> --
>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]atasyndrome.
>>> com
>>>
>>
>>
>>
>> --
>> *Note that I'm no longer using my Yahoo! email address. Please email me at
>> [EMAIL PROTECTED] going forward.*
>>
