Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Turn off Speculative Execution in a UDF?


Copy link to this message
-
Re: Turn off Speculative Execution in a UDF?
Hi all,

I am a little unclear about which speculative execution you must disable.
 What phase do storage functions run at map or reduce?

I've always just done both.
set mapred.reduce.tasks.speculative.execution false
set mapred.map.tasks.speculative.execution false

Thanks for any hints!

On Fri, Mar 2, 2012 at 4:18 PM, Bill Graham <[EMAIL PROTECTED]> wrote:

> I tried to test turning this off in the setStoreLocation method but without
> that change I wasn't able to get a job to run with SE happening. As a
> result I can't verify that the setting is doing anything. Russell, if you
> can reproduce SE I'd be curious to hear if you could turn it off in
> setStoreLocation.
>
> On Fri, Mar 2, 2012 at 2:40 PM, Russell Jurney <[EMAIL PROTECTED]
> >wrote:
>
> > I thought it was too late in the workflow to do this, but it would be
> > really cool if you could.  I don't think to think about MapReduce much
> when
> > I Pig, except to group my scripts by jobs... so this was a surprise for
> me.
> >  Made sense once i thought of it.  But it was a surprise.
> >
> >
> > On Fri, Mar 2, 2012 at 1:49 PM, Bill Graham <[EMAIL PROTECTED]>
> wrote:
> >
> >> I was also curious about this and will try it, but my initial thought
> was
> >> that at that point it might be tool late in the workflow of the job.
> I'll
> >> give it a shot and report back.
> >>
> >>
> >> On Fri, Mar 2, 2012 at 1:45 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > In a StoreFunc, you could do that when you get passed the jobconf,
> >> right?
> >> >
> >> > On Thu, Mar 1, 2012 at 9:37 PM, Bill Graham <[EMAIL PROTECTED]>
> >> wrote:
> >> > > I don't think so. We just do it in the pig script before using the
> >> store
> >> > > func:
> >> > >
> >> > > SET mapred.map.tasks.speculative.execution false
> >> > >
> >> > >
> >> > > On Thu, Mar 1, 2012 at 4:19 PM, Russell Jurney <
> >> [EMAIL PROTECTED]
> >> > >wrote:
> >> > >
> >> > >> Can you turn off speculative execution in a StoreFunc?  I believe
> it
> >> is
> >> > >> leading to duplicates in MongoStorage();
> >> > >>
> >> > >> --
> >> > >> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> >> > >> datasyndrome.com
> >> > >>
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > *Note that I'm no longer using my Yahoo! email address. Please email
> >> me
> >> > at
> >> > > [EMAIL PROTECTED] going forward.*
> >> >
> >>
> >>
> >>
> >> --
> >> *Note that I'm no longer using my Yahoo! email address. Please email me
> at
> >> [EMAIL PROTECTED] going forward.*
> >>
> >
> >
> >
> > --
> > Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]atasyndrome.
> > com
> >
>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> [EMAIL PROTECTED] going forward.*
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB