Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase bulk loader doing speculative execution when it set to false in mapred-site.xml


Copy link to this message
-
Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml
Hi Doug,

Yes, that's why i had set that property as false in my mapred-site.xml.
But, to my surprise i didnt know that setting that property would be
useless for Hadoop jobs unless the mapred-site.xml is in classpath. The
idea of client side property is a little confusing to me at present since
there is no proper nomenclature for client side properties at present.
Thanks for your reply.

~Anil

On Fri, Mar 30, 2012 at 3:26 PM, Doug Meil <[EMAIL PROTECTED]>wrote:

>
> Speculative execution is on by default in Hadoop.  One of the Performance
> recommendations in the Hbase RefGuide is to turn it off.
>
>
>
>
>
> On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:
>
> >Well that's not an HBase configuration, that's Hadoop. I'm not sure if
> >this is listed anywhere, maybe in the book.
> >
> >BTW usually HBase has a "client" somewhere in the same to indicate
> >it's client side.
> >
> >J-D
> >
> >On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[EMAIL PROTECTED]> wrote:
> >> Thanks for the quick reply, Jean. Is there any link where i can find the
> >> name of all client-side configuration for HBase?
> >>
> >> ~Anil
> >>
> >> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans
> >><[EMAIL PROTECTED]>wrote:
> >>
> >>> This is a client-side configuration so if your mapred-site.xml is
> >>> _not_ on your classpath when you start the bulk load, it's not going
> >>> to pick it up. So either have that file on your classpath, or put it
> >>> in whatever other configuration file you have.
> >>>
> >>> J-D
> >>>
> >>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]>
> >>>wrote:
> >>> > Hi All,
> >>> >
> >>> > I am using cdh3u2. I ran HBase bulk loading with property
> >>> > "mapred.reduce.tasks.speculative.execution" set to "false" in
> >>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job
> >>>and
> >>> > after short analysis i realized that these jobs are killed because
> >>> another
> >>> > worker node completed the task, hence it means that speculative
> >>>execution
> >>> > is still on. Why the HBase Bulk loader is doing speculative execution
> >>> when
> >>> > i have set it to false in mapred-site.xml? Please let me know if i am
> >>> > missing something over here.
> >>> >
> >>> > --
> >>> > Thanks & Regards,
> >>> > Anil Gupta
> >>>
> >>
> >>
> >>
> >> --
> >> Thanks & Regards,
> >> Anil Gupta
> >
>
>
>
--
Thanks & Regards,
Anil Gupta