Hi,

I've created patch for MAPREDUCE-4502. Now, I confirmed that it works
well for usual case, and I also added code to handle MapTask failure.

As a next step, I need to add test code against MapTask failure.

So I have questions:
Is there fault injection in MapReduce testing framework?
If the answer is negative, do you have any ideas to test it?

Thanks,
OZAWA Tsuyoshi

Hadoop-1 includes framework called Herriot that would allow you to develop
on-the-cluster FI system tests. However, because of the some timing, it hasn't
been hooked into the maven build system Hadoop-2 branches.

Basically, I see two way of doing what you need to do here:
  - wait until the Herriot is integrated back (that might take a while,
    actually)
  - go along with MOP using Groovy and develop a cluster test for your
    feature. MOP won't require pretty much anything but a groovy jar to be
    added to the classpath of the java process(es) in question. With it in
    place you can instrument anything you want the way you need during the
    application bootstrap. In fact, I think Herriot would be better off with
    that approach instead of initial AspectJ build-time mechanism.

Hope it helps,
  Cos

On Wed, Jan 16, 2013 at 02:19AM, Tsuyoshi OZAWA wrote:
> Hi,
>
> I've created patch for MAPREDUCE-4502. Now, I confirmed that it works
> well for usual case, and I also added code to handle MapTask failure.
>
> As a next step, I need to add test code against MapTask failure.
>
> So I have questions:
> Is there fault injection in MapReduce testing framework?
> If the answer is negative, do you have any ideas to test it?
>
> Thanks,
> OZAWA Tsuyoshi

Thanks for your comment.
Your comment is helpful for me.

I'd like to go with 2nd approach - MOP with Groovy. In that case, how
can I add test code to the trunk?
Is it acceptable for Hadoop project to add test code written in groovy?

Thanks,
Tsuyoshi

On Wed, Jan 16, 2013 at 12:13 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> Hadoop-1 includes framework called Herriot that would allow you to develop
> on-the-cluster FI system tests. However, because of the some timing, it hasn't
> been hooked into the maven build system Hadoop-2 branches.
>
> Basically, I see two way of doing what you need to do here:
>   - wait until the Herriot is integrated back (that might take a while,
>     actually)
>   - go along with MOP using Groovy and develop a cluster test for your
>     feature. MOP won't require pretty much anything but a groovy jar to be
>     added to the classpath of the java process(es) in question. With it in
>     place you can instrument anything you want the way you need during the
>     application bootstrap. In fact, I think Herriot would be better off with
>     that approach instead of initial AspectJ build-time mechanism.
>
> Hope it helps,
>   Cos
>
> On Wed, Jan 16, 2013 at 02:19AM, Tsuyoshi OZAWA wrote:
>> Hi,
>>
>> I've created patch for MAPREDUCE-4502. Now, I confirmed that it works
>> well for usual case, and I also added code to handle MapTask failure.
>>
>> As a next step, I need to add test code against MapTask failure.
>>
>> So I have questions:
>> Is there fault injection in MapReduce testing framework?
>> If the answer is negative, do you have any ideas to test it?
>>
>> Thanks,
>> OZAWA Tsuyoshi

--
OZAWA Tsuyoshi

On Wed, Jan 16, 2013 at 01:18PM, Tsuyoshi OZAWA wrote:
> Thanks for your comment.
> Your comment is helpful for me.
>
> I'd like to go with 2nd approach - MOP with Groovy. In that case, how
> can I add test code to the trunk?

You go with a additional patch for the test and the test-time dependencies
added.

> Is it acceptable for Hadoop project to add test code written in groovy?

Groovy is a Java+ Having Groovy tests won't require any massive build up of
the infrastructure - just an extra jar file, that will be visible in the test
scope only. While there are might different opinions in the community, as
f course, I don't see any real issues with that approach.

Cos

> Thanks,
> Tsuyoshi
>
> On Wed, Jan 16, 2013 at 12:13 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> > Hadoop-1 includes framework called Herriot that would allow you to develop
> > on-the-cluster FI system tests. However, because of the some timing, it hasn't
> > been hooked into the maven build system Hadoop-2 branches.
> >
> > Basically, I see two way of doing what you need to do here:
> >   - wait until the Herriot is integrated back (that might take a while,
> >     actually)
> >   - go along with MOP using Groovy and develop a cluster test for your
> >     feature. MOP won't require pretty much anything but a groovy jar to be
> >     added to the classpath of the java process(es) in question. With it in
> >     place you can instrument anything you want the way you need during the
> >     application bootstrap. In fact, I think Herriot would be better off with
> >     that approach instead of initial AspectJ build-time mechanism.
> >
> > Hope it helps,
> >   Cos
> >
> > On Wed, Jan 16, 2013 at 02:19AM, Tsuyoshi OZAWA wrote:
> >> Hi,
> >>
> >> I've created patch for MAPREDUCE-4502. Now, I confirmed that it works
> >> well for usual case, and I also added code to handle MapTask failure.
> >>
> >> As a next step, I need to add test code against MapTask failure.
> >>
> >> So I have questions:
> >> Is there fault injection in MapReduce testing framework?
> >> If the answer is negative, do you have any ideas to test it?
> >>
> >> Thanks,
> >> OZAWA Tsuyoshi
>
>
>
> --
> OZAWA Tsuyoshi

> You go with a additional patch for the test and the test-time dependencies
added.

I see, I've understood its simplicity may be acceptable.
I'll try it.

Tsuyoshi

On Wed, Jan 16, 2013 at 1:32 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> On Wed, Jan 16, 2013 at 01:18PM, Tsuyoshi OZAWA wrote:
>> Thanks for your comment.
>> Your comment is helpful for me.
>>
>> I'd like to go with 2nd approach - MOP with Groovy. In that case, how
>> can I add test code to the trunk?
>
> You go with a additional patch for the test and the test-time dependencies
> added.
>
>> Is it acceptable for Hadoop project to add test code written in groovy?
>
> Groovy is a Java+ Having Groovy tests won't require any massive build up of
> the infrastructure - just an extra jar file, that will be visible in the test
> scope only. While there are might different opinions in the community, as
> f course, I don't see any real issues with that approach.
>
> Cos
>
>> Thanks,
>> Tsuyoshi
>>
>> On Wed, Jan 16, 2013 at 12:13 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>> > Hadoop-1 includes framework called Herriot that would allow you to develop
>> > on-the-cluster FI system tests. However, because of the some timing, it hasn't
>> > been hooked into the maven build system Hadoop-2 branches.
>> >
>> > Basically, I see two way of doing what you need to do here:
>> >   - wait until the Herriot is integrated back (that might take a while,
>> >     actually)
>> >   - go along with MOP using Groovy and develop a cluster test for your
>> >     feature. MOP won't require pretty much anything but a groovy jar to be
>> >     added to the classpath of the java process(es) in question. With it in
>> >     place you can instrument anything you want the way you need during the
>> >     application bootstrap. In fact, I think Herriot would be better off with
>> >     that approach instead of initial AspectJ build-time mechanism.
>> >
>> > Hope it helps,
>> >   Cos
>> >
>> > On Wed, Jan 16, 2013 at 02:19AM, Tsuyoshi OZAWA wrote:
>> >> Hi,
>> >>
>> >> I've created patch for MAPREDUCE-4502. Now, I confirmed that it works
>> >> well for usual case, and I also added code to handle MapTask failure.
>> >>
>> >> As a next step, I need to add test code against MapTask failure.
>> >>
>> >> So I have questions:
>> >> Is there fault injection in MapReduce testing framework?
>> >> If the answer is negative, do you have any ideas to test it?
>> >>
>> >> Thanks,
>> >> OZAWA Tsuyoshi
>>
>>
>>
>> --
>> OZAWA Tsuyoshi

--
OZAWA Tsuyoshi

I confirmed that tests code with Groovy works well.
I'll throw patch when I finish to write my test case.

Thanks,
Tsuyoshi

On Wed, Jan 16, 2013 at 2:21 PM, Tsuyoshi OZAWA
<[EMAIL PROTECTED]> wrote:
>> You go with a additional patch for the test and the test-time dependencies
> added.
>
> I see, I've understood its simplicity may be acceptable.
> I'll try it.
>
> Tsuyoshi
>
> On Wed, Jan 16, 2013 at 1:32 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>> On Wed, Jan 16, 2013 at 01:18PM, Tsuyoshi OZAWA wrote:
>>> Thanks for your comment.
>>> Your comment is helpful for me.
>>>
>>> I'd like to go with 2nd approach - MOP with Groovy. In that case, how
>>> can I add test code to the trunk?
>>
>> You go with a additional patch for the test and the test-time dependencies
>> added.
>>
>>> Is it acceptable for Hadoop project to add test code written in groovy?
>>
>> Groovy is a Java+ Having Groovy tests won't require any massive build up of
>> the infrastructure - just an extra jar file, that will be visible in the test
>> scope only. While there are might different opinions in the community, as
>> f course, I don't see any real issues with that approach.
>>
>> Cos
>>
>>> Thanks,
>>> Tsuyoshi
>>>
>>> On Wed, Jan 16, 2013 at 12:13 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>>> > Hadoop-1 includes framework called Herriot that would allow you to develop
>>> > on-the-cluster FI system tests. However, because of the some timing, it hasn't
>>> > been hooked into the maven build system Hadoop-2 branches.
>>> >
>>> > Basically, I see two way of doing what you need to do here:
>>> >   - wait until the Herriot is integrated back (that might take a while,
>>> >     actually)
>>> >   - go along with MOP using Groovy and develop a cluster test for your
>>> >     feature. MOP won't require pretty much anything but a groovy jar to be
>>> >     added to the classpath of the java process(es) in question. With it in
>>> >     place you can instrument anything you want the way you need during the
>>> >     application bootstrap. In fact, I think Herriot would be better off with
>>> >     that approach instead of initial AspectJ build-time mechanism.
>>> >
>>> > Hope it helps,
>>> >   Cos
>>> >
>>> > On Wed, Jan 16, 2013 at 02:19AM, Tsuyoshi OZAWA wrote:
>>> >> Hi,
>>> >>
>>> >> I've created patch for MAPREDUCE-4502. Now, I confirmed that it works
>>> >> well for usual case, and I also added code to handle MapTask failure.
>>> >>
>>> >> As a next step, I need to add test code against MapTask failure.
>>> >>
>>> >> So I have questions:
>>> >> Is there fault injection in MapReduce testing framework?
>>> >> If the answer is negative, do you have any ideas to test it?
>>> >>
>>> >> Thanks,
>>> >> OZAWA Tsuyoshi
>>>
>>>
>>>
>>> --
>>> OZAWA Tsuyoshi
>
>
>
> --
> OZAWA Tsuyoshi

--
OZAWA Tsuyoshi