Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> can't disable speculative execution?


Copy link to this message
-
Re: can't disable speculative execution?
yes, let me try that

changing the max mapper slot actually requires changing the hadoop config,
since I just found that
it's "final" param

On Wed, Jul 11, 2012 at 10:05 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Your problem is more from the fact that you are running > 1 map slot
> per TT, and multiple mappers are getting run at the same time, all
> trying to bind to the same port. Limit your TT's max map tasks to 1
> when you're relying on such techniques to debug, or use the
> LocalJobRunner/Apache MRUnit instead.
>
> On Thu, Jul 12, 2012 at 9:16 AM, Yang <[EMAIL PROTECTED]> wrote:
> > I set the following params to be false in my pig script (0.10.0)
> >
> > SET mapred.map.tasks.speculative.execution false;
> > SET mapred.reduce.tasks.speculative.execution false;
> >
> >
> > I also verified in the jobtracker UI in the job.xml that they are indeed
> > set correctly.
> >
> > when the job finished, jobtracker UI shows that there is only one attempt
> > for each task (in fact I have only 1 task too).
> >
> > but when I went to the tasktracker node, looked under the
> > /var/log/hadoop/userlogs/job_id_here/
> > dir , there are 3 attempts dir ,
> >  job_201207111710_0024 # ls
> > attempt_201207111710_0024_m_000000_0
>  attempt_201207111710_0024_m_000001_0
> >  attempt_201207111710_0024_m_000002_0  job-acls.xml
> >
> > so 3 attempts were indeed fired ??
> >
> > I have to get this controlled correctly because I'm trying to debug the
> > mappers through eclipse,
> > but if more than 1 mapper process is fired, they all try to connect to
> the
> > same debugger port, and the end result is that nobody is able to
> > hook to the debugger.
> >
> >
> > Thanks
> > Yang
>
>
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB