Hi All,

I am using cdh3u2. I ran HBase bulk loading with property
"mapred.reduce.tasks.speculative.execution" set to "false" in
mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job and
after short analysis i realized that these jobs are killed because another
worker node completed the task, hence it means that speculative execution
is still on. Why the HBase Bulk loader is doing speculative execution when
i have set it to false in mapred-site.xml? Please let me know if i am
missing something over here.

--
Thanks & Regards,
Anil Gupta

This is a client-side configuration so if your mapred-site.xml is
_not_ on your classpath when you start the bulk load, it's not going
to pick it up. So either have that file on your classpath, or put it
in whatever other configuration file you have.

J-D

On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]> wrote:
> Hi All,
>
> I am using cdh3u2. I ran HBase bulk loading with property
> "mapred.reduce.tasks.speculative.execution" set to "false" in
> mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job and
> after short analysis i realized that these jobs are killed because another
> worker node completed the task, hence it means that speculative execution
> is still on. Why the HBase Bulk loader is doing speculative execution when
> i have set it to false in mapred-site.xml? Please let me know if i am
> missing something over here.
>
> --
> Thanks & Regards,
> Anil Gupta

Thanks for the quick reply, Jean. Is there any link where i can find the
name of all client-side configuration for HBase?

~Anil

On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> This is a client-side configuration so if your mapred-site.xml is
> _not_ on your classpath when you start the bulk load, it's not going
> to pick it up. So either have that file on your classpath, or put it
> in whatever other configuration file you have.
>
> J-D
>
> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]> wrote:
> > Hi All,
> >
> > I am using cdh3u2. I ran HBase bulk loading with property
> > "mapred.reduce.tasks.speculative.execution" set to "false" in
> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job and
> > after short analysis i realized that these jobs are killed because
> another
> > worker node completed the task, hence it means that speculative execution
> > is still on. Why the HBase Bulk loader is doing speculative execution
> when
> > i have set it to false in mapred-site.xml? Please let me know if i am
> > missing something over here.
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>

--
Thanks & Regards,
Anil Gupta

Well that's not an HBase configuration, that's Hadoop. I'm not sure if
this is listed anywhere, maybe in the book.

BTW usually HBase has a "client" somewhere in the same to indicate
it's client side.

J-D

On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[EMAIL PROTECTED]> wrote:
> Thanks for the quick reply, Jean. Is there any link where i can find the
> name of all client-side configuration for HBase?
>
> ~Anil
>
> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> This is a client-side configuration so if your mapred-site.xml is
>> _not_ on your classpath when you start the bulk load, it's not going
>> to pick it up. So either have that file on your classpath, or put it
>> in whatever other configuration file you have.
>>
>> J-D
>>
>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]> wrote:
>> > Hi All,
>> >
>> > I am using cdh3u2. I ran HBase bulk loading with property
>> > "mapred.reduce.tasks.speculative.execution" set to "false" in
>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job and
>> > after short analysis i realized that these jobs are killed because
>> another
>> > worker node completed the task, hence it means that speculative execution
>> > is still on. Why the HBase Bulk loader is doing speculative execution
>> when
>> > i have set it to false in mapred-site.xml? Please let me know if i am
>> > missing something over here.
>> >
>> > --
>> > Thanks & Regards,
>> > Anil Gupta
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta

Speculative execution is on by default in Hadoop.  One of the Performance
recommendations in the Hbase RefGuide is to turn it off.

On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:

>Well that's not an HBase configuration, that's Hadoop. I'm not sure if
>this is listed anywhere, maybe in the book.
>
>BTW usually HBase has a "client" somewhere in the same to indicate
>it's client side.
>
>J-D
>
>On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[EMAIL PROTECTED]> wrote:
>> Thanks for the quick reply, Jean. Is there any link where i can find the
>> name of all client-side configuration for HBase?
>>
>> ~Anil
>>
>> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans
>><[EMAIL PROTECTED]>wrote:
>>
>>> This is a client-side configuration so if your mapred-site.xml is
>>> _not_ on your classpath when you start the bulk load, it's not going
>>> to pick it up. So either have that file on your classpath, or put it
>>> in whatever other configuration file you have.
>>>
>>> J-D
>>>
>>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]>
>>>wrote:
>>> > Hi All,
>>> >
>>> > I am using cdh3u2. I ran HBase bulk loading with property
>>> > "mapred.reduce.tasks.speculative.execution" set to "false" in
>>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job
>>>and
>>> > after short analysis i realized that these jobs are killed because
>>> another
>>> > worker node completed the task, hence it means that speculative
>>>execution
>>> > is still on. Why the HBase Bulk loader is doing speculative execution
>>> when
>>> > i have set it to false in mapred-site.xml? Please let me know if i am
>>> > missing something over here.
>>> >
>>> > --
>>> > Thanks & Regards,
>>> > Anil Gupta
>>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>

Hi Doug,

Yes, that's why i had set that property as false in my mapred-site.xml.
But, to my surprise i didnt know that setting that property would be
useless for Hadoop jobs unless the mapred-site.xml is in classpath. The
idea of client side property is a little confusing to me at present since
there is no proper nomenclature for client side properties at present.
Thanks for your reply.

~Anil

On Fri, Mar 30, 2012 at 3:26 PM, Doug Meil <[EMAIL PROTECTED]>wrote:

>
> Speculative execution is on by default in Hadoop.  One of the Performance
> recommendations in the Hbase RefGuide is to turn it off.
>
>
>
>
>
> On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:
>
> >Well that's not an HBase configuration, that's Hadoop. I'm not sure if
> >this is listed anywhere, maybe in the book.
> >
> >BTW usually HBase has a "client" somewhere in the same to indicate
> >it's client side.
> >
> >J-D
> >
> >On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[EMAIL PROTECTED]> wrote:
> >> Thanks for the quick reply, Jean. Is there any link where i can find the
> >> name of all client-side configuration for HBase?
> >>
> >> ~Anil
> >>
> >> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans
> >><[EMAIL PROTECTED]>wrote:
> >>
> >>> This is a client-side configuration so if your mapred-site.xml is
> >>> _not_ on your classpath when you start the bulk load, it's not going
> >>> to pick it up. So either have that file on your classpath, or put it
> >>> in whatever other configuration file you have.
> >>>
> >>> J-D
> >>>
> >>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]>
> >>>wrote:
> >>> > Hi All,
> >>> >
> >>> > I am using cdh3u2. I ran HBase bulk loading with property
> >>> > "mapred.reduce.tasks.speculative.execution" set to "false" in
> >>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job
> >>>and
> >>> > after short analysis i realized that these jobs are killed because
> >>> another
> >>> > worker node completed the task, hence it means that speculative
> >>>execution
> >>> > is still on. Why the HBase Bulk loader is doing speculative execution
> >>> when
> >>> > i have set it to false in mapred-site.xml? Please let me know if i am
> >>> > missing something over here.
> >>> >
> >>> > --
> >>> > Thanks & Regards,
> >>> > Anil Gupta
> >>>
> >>
> >>
> >>
> >> --
> >> Thanks & Regards,
> >> Anil Gupta
> >
>
>
>
--
Thanks & Regards,
Anil Gupta

Anil,

You can also disable speculative execution on a per-job basis. See
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setMapSpeculativeExecution(boolean)
(Which is why it is called a client-sided property - it applies
per-job).

If HBase strongly recommends turning it off, HBase should also, by
default, turn it off for its own offered jobs?

On Sat, Mar 31, 2012 at 4:02 AM, anil gupta <[EMAIL PROTECTED]> wrote:
> Hi Doug,
>
> Yes, that's why i had set that property as false in my mapred-site.xml.
> But, to my surprise i didnt know that setting that property would be
> useless for Hadoop jobs unless the mapred-site.xml is in classpath. The
> idea of client side property is a little confusing to me at present since
> there is no proper nomenclature for client side properties at present.
> Thanks for your reply.
>
> ~Anil
>
> On Fri, Mar 30, 2012 at 3:26 PM, Doug Meil <[EMAIL PROTECTED]>wrote:
>
>>
>> Speculative execution is on by default in Hadoop.  One of the Performance
>> recommendations in the Hbase RefGuide is to turn it off.
>>
>>
>>
>>
>>
>> On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:
>>
>> >Well that's not an HBase configuration, that's Hadoop. I'm not sure if
>> >this is listed anywhere, maybe in the book.
>> >
>> >BTW usually HBase has a "client" somewhere in the same to indicate
>> >it's client side.
>> >
>> >J-D
>> >
>> >On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[EMAIL PROTECTED]> wrote:
>> >> Thanks for the quick reply, Jean. Is there any link where i can find the
>> >> name of all client-side configuration for HBase?
>> >>
>> >> ~Anil
>> >>
>> >> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans
>> >><[EMAIL PROTECTED]>wrote:
>> >>
>> >>> This is a client-side configuration so if your mapred-site.xml is
>> >>> _not_ on your classpath when you start the bulk load, it's not going
>> >>> to pick it up. So either have that file on your classpath, or put it
>> >>> in whatever other configuration file you have.
>> >>>
>> >>> J-D
>> >>>
>> >>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]>
>> >>>wrote:
>> >>> > Hi All,
>> >>> >
>> >>> > I am using cdh3u2. I ran HBase bulk loading with property
>> >>> > "mapred.reduce.tasks.speculative.execution" set to "false" in
>> >>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job
>> >>>and
>> >>> > after short analysis i realized that these jobs are killed because
>> >>> another
>> >>> > worker node completed the task, hence it means that speculative
>> >>>execution
>> >>> > is still on. Why the HBase Bulk loader is doing speculative execution
>> >>> when
>> >>> > i have set it to false in mapred-site.xml? Please let me know if i am
>> >>> > missing something over here.
>> >>> >
>> >>> > --
>> >>> > Thanks & Regards,
>> >>> > Anil Gupta
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Thanks & Regards,
>> >> Anil Gupta
>> >
>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta

--
Harsh J

Hash
I think you have a good point here - It is a good practice that the
utilities given from HBase also adapt HBase own recommendations.
For example the RowCounter
(org.apache.hadoop.hbase.mapreduce.RowCounter.java) utility is neither
setting the speculative execution to 'false', nor the scan cache to a value
different to zero, which is not according to the hbase book/ref guide  (
http://hbase.apache.org/book.html#mapreduce )

Mikael.S

On Sat, Mar 31, 2012 at 7:54 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Anil,
>
> You can also disable speculative execution on a per-job basis. See
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setMapSpeculativeExecution(boolean)
> (Which is why it is called a client-sided property - it applies
> per-job).
>
> If HBase strongly recommends turning it off, HBase should also, by
> default, turn it off for its own offered jobs?
>
> On Sat, Mar 31, 2012 at 4:02 AM, anil gupta <[EMAIL PROTECTED]> wrote:
> > Hi Doug,
> >
> > Yes, that's why i had set that property as false in my mapred-site.xml.
> > But, to my surprise i didnt know that setting that property would be
> > useless for Hadoop jobs unless the mapred-site.xml is in classpath. The
> > idea of client side property is a little confusing to me at present since
> > there is no proper nomenclature for client side properties at present.
> > Thanks for your reply.
> >
> > ~Anil
> >
> > On Fri, Mar 30, 2012 at 3:26 PM, Doug Meil <
> [EMAIL PROTECTED]>wrote:
> >
> >>
> >> Speculative execution is on by default in Hadoop.  One of the
> Performance
> >> recommendations in the Hbase RefGuide is to turn it off.
> >>
> >>
> >>
> >>
> >>
> >> On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:
> >>
> >> >Well that's not an HBase configuration, that's Hadoop. I'm not sure if
> >> >this is listed anywhere, maybe in the book.
> >> >
> >> >BTW usually HBase has a "client" somewhere in the same to indicate
> >> >it's client side.
> >> >
> >> >J-D
> >> >
> >> >On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[EMAIL PROTECTED]>
> wrote:
> >> >> Thanks for the quick reply, Jean. Is there any link where i can find
> the
> >> >> name of all client-side configuration for HBase?
> >> >>
> >> >> ~Anil
> >> >>
> >> >> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans
> >> >><[EMAIL PROTECTED]>wrote:
> >> >>
> >> >>> This is a client-side configuration so if your mapred-site.xml is
> >> >>> _not_ on your classpath when you start the bulk load, it's not going
> >> >>> to pick it up. So either have that file on your classpath, or put it
> >> >>> in whatever other configuration file you have.
> >> >>>
> >> >>> J-D
> >> >>>
> >> >>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[EMAIL PROTECTED]>
> >> >>>wrote:
> >> >>> > Hi All,
> >> >>> >
> >> >>> > I am using cdh3u2. I ran HBase bulk loading with property
> >> >>> > "mapred.reduce.tasks.speculative.execution" set to "false" in
> >> >>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading
> job
> >> >>>and
> >> >>> > after short analysis i realized that these jobs are killed because
> >> >>> another
> >> >>> > worker node completed the task, hence it means that speculative
> >> >>>execution
> >> >>> > is still on. Why the HBase Bulk loader is doing speculative
> execution
> >> >>> when
> >> >>> > i have set it to false in mapred-site.xml? Please let me know if
> i am
> >> >>> > missing something over here.
> >> >>> >
> >> >>> > --
> >> >>> > Thanks & Regards,
> >> >>> > Anil Gupta
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Thanks & Regards,
> >> >> Anil Gupta
> >> >
> >>
> >>
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
>
>
> --
> Harsh J
>

--
Mikael.S

It might work to set the property as final on the server side, so that clients can't override it:

<property>
<name>mapred.reduce.tasks.speculative.execution</name>
<value>false</value>
<final>true</final>
<description>If true, then multiple instances of some reduce tasks
may be executed in parallel.
</description>
</property>

Seems like having it set as final at the JT would prevent speculative execution no matter what the client says, but I could be mistaken.

Sandy

> -----Original Message-----
> From: anil gupta [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 30, 2012 14:53
> To: [EMAIL PROTECTED]
> Subject: HBase bulk loader doing speculative execution when it set to false in
> mapred-site.xml
>
> Hi All,
>
> I am using cdh3u2. I ran HBase bulk loading with property
> "mapred.reduce.tasks.speculative.execution" set to "false" in mapred-
> site.xml. Still, i can see 6 killed task in Bulk Loading job and after short analysis
> i realized that these jobs are killed because another worker node completed
> the task, hence it means that speculative execution is still on. Why the HBase
> Bulk loader is doing speculative execution when i have set it to false in
> mapred-site.xml? Please let me know if i am missing something over here.
>
> --
> Thanks & Regards,
> Anil Gupta

Thanks for the suggestion, Sandy. I wil let you know the outcome once i run
the job.

On Mon, Apr 2, 2012 at 3:26 PM, Sandy Pratt <[EMAIL PROTECTED]> wrote:

> It might work to set the property as final on the server side, so that
> clients can't override it:
>
>        <property>
>                <name>mapred.reduce.tasks.speculative.execution</name>
>                <value>false</value>
>                <final>true</final>
>                <description>If true, then multiple instances of some
> reduce tasks
>                        may be executed in parallel.
>                </description>
>        </property>
>
> Seems like having it set as final at the JT would prevent speculative
> execution no matter what the client says, but I could be mistaken.
>
> Sandy
>
> > -----Original Message-----
> > From: anil gupta [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, March 30, 2012 14:53
> > To: [EMAIL PROTECTED]
> > Subject: HBase bulk loader doing speculative execution when it set to
> false in
> > mapred-site.xml
> >
> > Hi All,
> >
> > I am using cdh3u2. I ran HBase bulk loading with property
> > "mapred.reduce.tasks.speculative.execution" set to "false" in mapred-
> > site.xml. Still, i can see 6 killed task in Bulk Loading job and after
> short analysis
> > i realized that these jobs are killed because another worker node
> completed
> > the task, hence it means that speculative execution is still on. Why the
> HBase
> > Bulk loader is doing speculative execution when i have set it to false in
> > mapred-site.xml? Please let me know if i am missing something over here.
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>

--
Thanks & Regards,
Anil Gupta