Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml


Copy link to this message
-
Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml
+[EMAIL PROTECTED]

Hi Harsh,

Thanks for the information.
Is there any way to differentiate between a client side property and
server-side property?or a Document which enlists whether a property is
server or client-side? Many times i have to speculate over this and try out
test runs.

Thanks,
Anil

On Fri, Mar 30, 2012 at 9:54 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Anil,
>
> You can also disable speculative execution on a per-job basis. See
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setMapSpeculativeExecution(boolean)
> (Which is why it is called a client-sided property - it applies
> per-job).
>
> If HBase strongly recommends turning it off, HBase should also, by
> default, turn it off for its own offered jobs?
>
> On Sat, Mar 31, 2012 at 4:02 AM, anil gupta <[EMAIL PROTECTED]> wrote:
> > Hi Doug,
> >
> > Yes, that's why i had set that property as false in my mapred-site.xml.
> > But, to my surprise i didnt know that setting that property would be
> > useless for Hadoop jobs unless the mapred-site.xml is in classpath. The
> > idea of client side property is a little confusing to me at present since
> > there is no proper nomenclature for client side properties at present.
> > Thanks for your reply.
> >
> > ~Anil
> >
> > On Fri, Mar 30, 2012 at 3:26 PM, Doug Meil <
> [EMAIL PROTECTED]>wrote:
> >
> >>
> >> Speculative execution is on by default in Hadoop.  One of the
> Performance
> >> recommendations in the Hbase RefGuide is to turn it off.
> >>
> >>
> >>
> >>
> >>
> >> On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:
> >>
> >> >Well that's not an HBase configuration, that's Hadoop. I'm not sure if
> >> >this is listed anywhere, maybe in the book.
> >> >
> >> >BTW usually HBase has a "client" somewhere in the same to indicate
> >> >it's client side.
> >> >
> >> >J-D
> >> >
> >> >On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[EMAIL PROTECTED]>
> wrote:
> >> >> Thanks for the quick reply, Jean. Is there any link where i can find
> the
> >> >> name of all client-side configuration for HBase?
> >> >>
> >> >> ~Anil
> >> >>
> >> >> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans
> >> >><[EMAIL PROTECTED]>wrote:
> >> >>
> >> >>> This is a client-side configuration so if your mapred-site.xml is
> >> >>> _not_ on your classpath when you start the bulk load, it's not going
> >> >>> to pick it up. So either have that file on your classpath, or put it
> >> >>> in whatever other configuration file you have.
> >> >>>
> >> >>> J-D
> >> >>>
> >> >>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]>
> >> >>>wrote:
> >> >>> > Hi All,
> >> >>> >
> >> >>> > I am using cdh3u2. I ran HBase bulk loading with property
> >> >>> > "mapred.reduce.tasks.speculative.execution" set to "false" in
> >> >>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading
> job
> >> >>>and
> >> >>> > after short analysis i realized that these jobs are killed because
> >> >>> another
> >> >>> > worker node completed the task, hence it means that speculative
> >> >>>execution
> >> >>> > is still on. Why the HBase Bulk loader is doing speculative
> execution
> >> >>> when
> >> >>> > i have set it to false in mapred-site.xml? Please let me know if
> i am
> >> >>> > missing something over here.
> >> >>> >
> >> >>> > --
> >> >>> > Thanks & Regards,
> >> >>> > Anil Gupta
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Thanks & Regards,
> >> >> Anil Gupta
> >> >
> >>
> >>
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
>
>
> --
> Harsh J
>

--
Thanks & Regards,
Anil Gupta
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB