Currently Jenkins QA builds don't support cross-project patches, since
they try to apply them to the hadoop-{common,hdfs,mapreduce}-project
tree, and reject any patches that are not strictly confined to that
tree. For changes that span projects (e.g. moving code from HDFS or MR
to Common, or build system changes) developers have to run test-patch
and tests manually, which is cumbersome.

I'd like to propose that we change the Common Jenkins build so that it
runs from the top-level
(http://svn.apache.org/repos/asf/hadoop/common/trunk). The HDFS and
MapReduce builds would be unchanged. This would have two implications:
i) all patches would have to be rooted at the top-level, and ii) the
unit tests for all projects would be run for Common changes.

I think i) is reasonable (and most patches are like this now) - and
it's easy to regenerate a patch that is rooted at the wrong level. For
ii), running all tests for a change in Common is probably a good thing
to do in general.

Thoughts?

Tom

+1

This JIRA has been sitting patch available for a few weeks:
https://issues.apache.org/jira/browse/HADOOP-7416

I don't think it's quite ready for prime time (it doesn't implement all the
features you described) but it'd be a good starting point if someone wanted
to take it over the finish line.

--
Aaron T. Myers
Software Engineer, Cloudera

On Mon, Apr 16, 2012 at 1:51 PM, Tom White <[EMAIL PROTECTED]> wrote:

> Currently Jenkins QA builds don't support cross-project patches, since
> they try to apply them to the hadoop-{common,hdfs,mapreduce}-project
> tree, and reject any patches that are not strictly confined to that
> tree. For changes that span projects (e.g. moving code from HDFS or MR
> to Common, or build system changes) developers have to run test-patch
> and tests manually, which is cumbersome.
>
> I'd like to propose that we change the Common Jenkins build so that it
> runs from the top-level
> (http://svn.apache.org/repos/asf/hadoop/common/trunk). The HDFS and
> MapReduce builds would be unchanged. This would have two implications:
> i) all patches would have to be rooted at the top-level, and ii) the
> unit tests for all projects would be run for Common changes.
>
> I think i) is reasonable (and most patches are like this now) - and
> it's easy to regenerate a patch that is rooted at the wrong level. For
> ii), running all tests for a change in Common is probably a good thing
> to do in general.
>
> Thoughts?
>
> Tom
>

+1,

I'd still go a bit further in the simplification of what testpatch does.

* patches must at root level (not even try to be smart)
* all testcases should always be run (else a change in hdfs could
affect yarn/tools but not be detected, or one in yarn affect tools)

Thxs.

Alejandro

On Mon, Apr 16, 2012 at 1:55 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:
> +1
>
> This JIRA has been sitting patch available for a few weeks:
> https://issues.apache.org/jira/browse/HADOOP-7416
>
> I don't think it's quite ready for prime time (it doesn't implement all the
> features you described) but it'd be a good starting point if someone wanted
> to take it over the finish line.
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
>
>
> On Mon, Apr 16, 2012 at 1:51 PM, Tom White <[EMAIL PROTECTED]> wrote:
>
>> Currently Jenkins QA builds don't support cross-project patches, since
>> they try to apply them to the hadoop-{common,hdfs,mapreduce}-project
>> tree, and reject any patches that are not strictly confined to that
>> tree. For changes that span projects (e.g. moving code from HDFS or MR
>> to Common, or build system changes) developers have to run test-patch
>> and tests manually, which is cumbersome.
>>
>> I'd like to propose that we change the Common Jenkins build so that it
>> runs from the top-level
>> (http://svn.apache.org/repos/asf/hadoop/common/trunk). The HDFS and
>> MapReduce builds would be unchanged. This would have two implications:
>> i) all patches would have to be rooted at the top-level, and ii) the
>> unit tests for all projects would be run for Common changes.
>>
>> I think i) is reasonable (and most patches are like this now) - and
>> it's easy to regenerate a patch that is rooted at the wrong level. For
>> ii), running all tests for a change in Common is probably a good thing
>> to do in general.
>>
>> Thoughts?
>>
>> Tom
>>

--
Alejandro

On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:

> * all testcases should always be run (else a change in hdfs could
> affect yarn/tools but not be detected, or one in yarn affect tools)
>

I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
dependent Hadoop sub-projects for every patch, but it will also
dramatically increase the time test-patch takes to run for any given patch.
In my experience, the vast majority of patches stand little chance of
breaking the dependent sub-projects, making this largely unnecessary and
thus a waste of time and Jenkins build slave resources.

--
Aaron T. Myers
Software Engineer, Cloudera

I agree with Aaron. Its going increase the test patch build timings
significantly which may not be very helpful

Im  -1 on this.

-Giri

On Mon, Apr 16, 2012 at 2:22 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:
>
>> * all testcases should always be run (else a change in hdfs could
>> affect yarn/tools but not be detected, or one in yarn affect tools)
>>
>
> I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
> dependent Hadoop sub-projects for every patch, but it will also
> dramatically increase the time test-patch takes to run for any given patch.
> In my experience, the vast majority of patches stand little chance of
> breaking the dependent sub-projects, making this largely unnecessary and
> thus a waste of time and Jenkins build slave resources.
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera

Giri,

I think Aaron was talking about not running all test cases for changes
to any project (e.g. HDFS and MapReduce). My proposal was to run all
the tests for any Common change. An HDFS change would only run HDFS
tests, and any MapReduce change would only run MapReduce tests.

Another thing I didn't mention was that currently Jenkins doesn't run
tests or apply patches for any changes in hadoop-tools, which would be
fixed by the change I'm suggesting.

Tom

On Tue, Apr 17, 2012 at 3:17 PM, Giridharan Kesavan
<[EMAIL PROTECTED]> wrote:
> I agree with Aaron. Its going increase the test patch build timings
> significantly which may not be very helpful
>
> Im  -1 on this.
>
> -Giri
>
>
>
> On Mon, Apr 16, 2012 at 2:22 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:
>> On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:
>>
>>> * all testcases should always be run (else a change in hdfs could
>>> affect yarn/tools but not be detected, or one in yarn affect tools)
>>>
>>
>> I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
>> dependent Hadoop sub-projects for every patch, but it will also
>> dramatically increase the time test-patch takes to run for any given patch.
>> In my experience, the vast majority of patches stand little chance of
>> breaking the dependent sub-projects, making this largely unnecessary and
>> thus a waste of time and Jenkins build slave resources.
>>
>> --
>> Aaron T. Myers
>> Software Engineer, Cloudera

Giri,

I agree that running ALL tests all time takes a lot of time
(personally I'd prefer we do this at the penalty of longer runs).

Still we have a problem to solve, we need to find a solution on
test-patch working for ALL maven modules, currently changes outside of
common/hdfs/mapred or cross-projects test-patch does not work.

So, how about the following approach:

* All patches must be at trunk/ level
* All patches do a full clean TARBALL creation without running testcases
* From the patch file we find out the maven modules and for those
modules we do javac-warns/javadoc-warns/findbugs/testcases

This would speed up test-patch runs and together with a nightly
jenkins jobs running ALL testcases would give a complete coverage.

Does this seem reasonable?

Thxs.

Alejandro

On Tue, Apr 17, 2012 at 3:31 PM, Tom White <[EMAIL PROTECTED]> wrote:
> Giri,
>
> I think Aaron was talking about not running all test cases for changes
> to any project (e.g. HDFS and MapReduce). My proposal was to run all
> the tests for any Common change. An HDFS change would only run HDFS
> tests, and any MapReduce change would only run MapReduce tests.
>
> Another thing I didn't mention was that currently Jenkins doesn't run
> tests or apply patches for any changes in hadoop-tools, which would be
> fixed by the change I'm suggesting.
>
> Tom
>
> On Tue, Apr 17, 2012 at 3:17 PM, Giridharan Kesavan
> <[EMAIL PROTECTED]> wrote:
>> I agree with Aaron. Its going increase the test patch build timings
>> significantly which may not be very helpful
>>
>> Im  -1 on this.
>>
>> -Giri
>>
>>
>>
>> On Mon, Apr 16, 2012 at 2:22 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:
>>> On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:
>>>
>>>> * all testcases should always be run (else a change in hdfs could
>>>> affect yarn/tools but not be detected, or one in yarn affect tools)
>>>>
>>>
>>> I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
>>> dependent Hadoop sub-projects for every patch, but it will also
>>> dramatically increase the time test-patch takes to run for any given patch.
>>> In my experience, the vast majority of patches stand little chance of
>>> breaking the dependent sub-projects, making this largely unnecessary and
>>> thus a waste of time and Jenkins build slave resources.
>>>
>>> --
>>> Aaron T. Myers
>>> Software Engineer, Cloudera

--
Alejandro

Alejandro,

On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
> Giri,
>
> I agree that running ALL tests all time takes a lot of time
> (personally I'd prefer we do this at the penalty of longer runs).
>
> Still we have a problem to solve, we need to find a solution on
> test-patch working for ALL maven modules, currently changes outside of
> common/hdfs/mapred or cross-projects test-patch does not work.
>
> So, how about the following approach:
>
> * All patches must be at trunk/ level
> * All patches do a full clean TARBALL creation without running testcases
> * From the patch file we find out the maven modules and for those
> modules we do javac-warns/javadoc-warns/findbugs/testcases

I like this approach of doing a clean tarball.
and doing the other checks ( javac warnings, javadoc warnings, findbug
warnings and release audit.)
for that specific module.

>
> This would speed up test-patch runs and together with a nightly
> jenkins jobs running ALL testcases would give a complete coverage.
>

test-patch and nightly jenkins jobs running ALL testcase?
could you pls explain this?

> Does this seem reasonable?
>
> Thxs.
>
> Alejandro
>
> On Tue, Apr 17, 2012 at 3:31 PM, Tom White <[EMAIL PROTECTED]> wrote:
>> Giri,
>>
>> I think Aaron was talking about not running all test cases for changes
>> to any project (e.g. HDFS and MapReduce). My proposal was to run all
>> the tests for any Common change. An HDFS change would only run HDFS
>> tests, and any MapReduce change would only run MapReduce tests.
>>
>> Another thing I didn't mention was that currently Jenkins doesn't run
>> tests or apply patches for any changes in hadoop-tools, which would be
>> fixed by the change I'm suggesting.
>>
>> Tom
>>
>> On Tue, Apr 17, 2012 at 3:17 PM, Giridharan Kesavan
>> <[EMAIL PROTECTED]> wrote:
>>> I agree with Aaron. Its going increase the test patch build timings
>>> significantly which may not be very helpful
>>>
>>> Im  -1 on this.
>>>
>>> -Giri
>>>
>>>
>>>
>>> On Mon, Apr 16, 2012 at 2:22 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:
>>>> On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> * all testcases should always be run (else a change in hdfs could
>>>>> affect yarn/tools but not be detected, or one in yarn affect tools)
>>>>>
>>>>
>>>> I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
>>>> dependent Hadoop sub-projects for every patch, but it will also
>>>> dramatically increase the time test-patch takes to run for any given patch.
>>>> In my experience, the vast majority of patches stand little chance of
>>>> breaking the dependent sub-projects, making this largely unnecessary and
>>>> thus a waste of time and Jenkins build slave resources.
>>>>
>>>> --
>>>> Aaron T. Myers
>>>> Software Engineer, Cloudera
>
>
>
> --
> Alejandro

Giri,

On Tue, Apr 17, 2012 at 8:36 PM, Giridharan Kesavan
<[EMAIL PROTECTED]> wrote:
> Alejandro,
>
> On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
>> Giri,
>>
>> I agree that running ALL tests all time takes a lot of time
>> (personally I'd prefer we do this at the penalty of longer runs).
>>
>> Still we have a problem to solve, we need to find a solution on
>> test-patch working for ALL maven modules, currently changes outside of
>> common/hdfs/mapred or cross-projects test-patch does not work.
>>
>> So, how about the following approach:
>>
>> * All patches must be at trunk/ level
>> * All patches do a full clean TARBALL creation without running testcases
>> * From the patch file we find out the maven modules and for those
>> modules we do javac-warns/javadoc-warns/findbugs/testcases
>
> I like this approach of doing a clean tarball.
> and doing the other checks ( javac warnings, javadoc warnings, findbug
> warnings and release audit.)
> for that specific module.
>

Great, the idea of the doing a clean tarball it to ensure that nothing
in the build/assembly is broken and that no API change breaks other
modules.

>>
>> This would speed up test-patch runs and together with a nightly
>> jenkins jobs running ALL testcases would give a complete coverage.
>>
>
> test-patch and nightly jenkins jobs running ALL testcase?
> could you pls explain this?

you want to verify there is no regression because of functional
changes in all modules, doing this once a day seems reasonable and
will help identify the culprit early on (that is why I said in my
original email my preference would be to run everything every time)

thxs

---
Alejandro

I have a patch at https://issues.apache.org/jira/browse/HADOOP-8308
which takes the approach that Alejandro outlined.

Cheers,
Tom

On Wed, Apr 18, 2012 at 10:11 AM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
> Giri,
>
> On Tue, Apr 17, 2012 at 8:36 PM, Giridharan Kesavan
> <[EMAIL PROTECTED]> wrote:
>> Alejandro,
>>
>> On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
>>> Giri,
>>>
>>> I agree that running ALL tests all time takes a lot of time
>>> (personally I'd prefer we do this at the penalty of longer runs).
>>>
>>> Still we have a problem to solve, we need to find a solution on
>>> test-patch working for ALL maven modules, currently changes outside of
>>> common/hdfs/mapred or cross-projects test-patch does not work.
>>>
>>> So, how about the following approach:
>>>
>>> * All patches must be at trunk/ level
>>> * All patches do a full clean TARBALL creation without running testcases
>>> * From the patch file we find out the maven modules and for those
>>> modules we do javac-warns/javadoc-warns/findbugs/testcases
>>
>> I like this approach of doing a clean tarball.
>> and doing the other checks ( javac warnings, javadoc warnings, findbug
>> warnings and release audit.)
>> for that specific module.
>>
>
> Great, the idea of the doing a clean tarball it to ensure that nothing
> in the build/assembly is broken and that no API change breaks other
> modules.
>
>>>
>>> This would speed up test-patch runs and together with a nightly
>>> jenkins jobs running ALL testcases would give a complete coverage.
>>>
>>
>> test-patch and nightly jenkins jobs running ALL testcase?
>> could you pls explain this?
>
> you want to verify there is no regression because of functional
> changes in all modules, doing this once a day seems reasonable and
> will help identify the culprit early on (that is why I said in my
> original email my preference would be to run everything every time)
>
> thxs
>
> ---
> Alejandro

On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>wrote:

> * All patches must be at trunk/ level
> * All patches do a full clean TARBALL creation without running testcases
> * From the patch file we find out the maven modules and for those
> modules we do javac-warns/javadoc-warns/findbugs/testcases
>

+1, I like this proposal a lot. Seems to do a good job of balancing the
need to check for dependent sub-project breakage with the desire to not
unnecessarily inflate the run time of test-patch.

--
Aaron T. Myers
Software Engineer, Cloudera

Dne 18.4.2012 0:17, Giridharan Kesavan napsal(a):
> I agree with Aaron. Its going increase the test patch build timings
> significantly which may not be very helpful
There are 9 build machines dedicated to hadoop. I would like that
automatic checking can finally test my patches like
MAPREDUCE-4116 <https://issues.apache.org/jira/browse/MAPREDUCE-4116>,
HADOOP-8317 <https://issues.apache.org/jira/browse/HADOOP-8317>,
HADOOP-8268. <https://issues.apache.org/jira/browse/HADOOP-8268>

Just to be clear, the HADOOP-8308 patch is selective about which tests
it runs, so it doesn't increase build run times. Unless there are any
objections, I'd like to commit the patch and switch Jenkins to run
from the top-level directory early next week.

Cheers,
Tom

On Fri, Apr 27, 2012 at 2:21 AM, Radim Kolar <[EMAIL PROTECTED]> wrote:
> Dne 18.4.2012 0:17, Giridharan Kesavan napsal(a):
>
>> I agree with Aaron. Its going increase the test patch build timings
>> significantly which may not be very helpful
>
> There are 9 build machines dedicated to hadoop. I would like that automatic
> checking can finally test my patches like
> MAPREDUCE-4116 <https://issues.apache.org/jira/browse/MAPREDUCE-4116>,
> HADOOP-8317 <https://issues.apache.org/jira/browse/HADOOP-8317>,
> HADOOP-8268. <https://issues.apache.org/jira/browse/HADOOP-8268>

PLEASEEEEE!!!!

On Fri, Apr 27, 2012 at 4:48 PM, Tom White <[EMAIL PROTECTED]> wrote:

> Just to be clear, the HADOOP-8308 patch is selective about which tests
> it runs, so it doesn't increase build run times. Unless there are any
> objections, I'd like to commit the patch and switch Jenkins to run
> from the top-level directory early next week.
>
> Cheers,
> Tom
>
> On Fri, Apr 27, 2012 at 2:21 AM, Radim Kolar <[EMAIL PROTECTED]> wrote:
> > Dne 18.4.2012 0:17, Giridharan Kesavan napsal(a):
> >
> >> I agree with Aaron. Its going increase the test patch build timings
> >> significantly which may not be very helpful
> >
> > There are 9 build machines dedicated to hadoop. I would like that
> automatic
> > checking can finally test my patches like
> > MAPREDUCE-4116 <https://issues.apache.org/jira/browse/MAPREDUCE-4116>,
> > HADOOP-8317 <https://issues.apache.org/jira/browse/HADOOP-8317>,
> > HADOOP-8268. <https://issues.apache.org/jira/browse/HADOOP-8268>
>

--
Alejandro