Brock Noland
2012-12-07, 16:11
Jarek Jarcec Cecho
2012-12-07, 16:26
Brock Noland
2012-12-07, 17:14
Jarek Jarcec Cecho
2012-12-07, 23:20
Mike Percy
2012-12-08, 08:39
Brock Noland
2012-12-20, 06:38
Jarek Jarcec Cecho
2012-12-20, 15:04
Brock Noland
2012-12-21, 00:25
-
[DISCUSS] Test on Patch AvailableBrock Noland 2012-12-07, 16:11
Hi,
I like how in the Hadoop project if you mark a patch "Patch Available"
jenkins applies the patch to trunk, runs the tests, and outputs the
result to the JIRA. Here is one example:
http://s.apache.org/qz
I think we should still run tests manually before commit. However,
this will often give a reviewer an indicator of whether the change
breaks a test without even having to download the code.
Thoughts?
Brock
I like how in the Hadoop project if you mark a patch "Patch Available"
jenkins applies the patch to trunk, runs the tests, and outputs the
result to the JIRA. Here is one example:
http://s.apache.org/qz
I think we should still run tests manually before commit. However,
this will often give a reviewer an indicator of whether the change
breaks a test without even having to download the code.
Thoughts?
Brock
+
Brock Noland 2012-12-07, 16:11
-
Re: [DISCUSS] Test on Patch AvailableJarek Jarcec Cecho 2012-12-07, 16:26
Hi Brock,
that is an excellent idea and I'm definitely in favour of that. In fact I wanted to propose something similar myself.
However I did not quite finished my own investigation yet. It seems that this pre commit hooks are done on projects that were mostly donated by Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named nodes have comment "Build machine for Hadoop project builds", so I'm not sure that they are available for other projects as well.
With removing those 9 hadoop notes, it's like 10 other nodes in total for entire Apache organization and I do know that sometimes the build queue might be quite huge. Having said that it might be good idea to ask Infra team if pre-commit hooks are available to all projects or only to projects that have "special hardware" available.
Jarcec
On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
> Hi,
>
> I like how in the Hadoop project if you mark a patch "Patch Available"
> jenkins applies the patch to trunk, runs the tests, and outputs the
> result to the JIRA. Here is one example:
>
> http://s.apache.org/qz
>
> I think we should still run tests manually before commit. However,
> this will often give a reviewer an indicator of whether the change
> breaks a test without even having to download the code.
>
> Thoughts?
>
> Brock
that is an excellent idea and I'm definitely in favour of that. In fact I wanted to propose something similar myself.
However I did not quite finished my own investigation yet. It seems that this pre commit hooks are done on projects that were mostly donated by Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named nodes have comment "Build machine for Hadoop project builds", so I'm not sure that they are available for other projects as well.
With removing those 9 hadoop notes, it's like 10 other nodes in total for entire Apache organization and I do know that sometimes the build queue might be quite huge. Having said that it might be good idea to ask Infra team if pre-commit hooks are available to all projects or only to projects that have "special hardware" available.
Jarcec
On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
> Hi,
>
> I like how in the Hadoop project if you mark a patch "Patch Available"
> jenkins applies the patch to trunk, runs the tests, and outputs the
> result to the JIRA. Here is one example:
>
> http://s.apache.org/qz
>
> I think we should still run tests manually before commit. However,
> this will often give a reviewer an indicator of whether the change
> breaks a test without even having to download the code.
>
> Thoughts?
>
> Brock
+
Jarek Jarcec Cecho 2012-12-07, 16:26
-
Re: [DISCUSS] Test on Patch AvailableBrock Noland 2012-12-07, 17:14
Hi,
Great to hear you were thinking the same thing! I have found the
script they use to do the checking:
https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
and I think we could probably implement something similar. I agree
that our build machines are much busier so I don't think we can use it
as a hard requirement for commit. The biggest open question in my mind
is how do they query JIRA for patches? Grab the correct patch off of
the JIRA? Schedule the job to poll for these things?
Brock
On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
> Hi Brock,
> that is an excellent idea and I'm definitely in favour of that. In fact I wanted to propose something similar myself.
>
> However I did not quite finished my own investigation yet. It seems that this pre commit hooks are done on projects that were mostly donated by Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named nodes have comment "Build machine for Hadoop project builds", so I'm not sure that they are available for other projects as well.
>
> With removing those 9 hadoop notes, it's like 10 other nodes in total for entire Apache organization and I do know that sometimes the build queue might be quite huge. Having said that it might be good idea to ask Infra team if pre-commit hooks are available to all projects or only to projects that have "special hardware" available.
>
> Jarcec
>
> On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
>> Hi,
>>
>> I like how in the Hadoop project if you mark a patch "Patch Available"
>> jenkins applies the patch to trunk, runs the tests, and outputs the
>> result to the JIRA. Here is one example:
>>
>> http://s.apache.org/qz
>>
>> I think we should still run tests manually before commit. However,
>> this will often give a reviewer an indicator of whether the change
>> breaks a test without even having to download the code.
>>
>> Thoughts?
>>
>> Brock
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Great to hear you were thinking the same thing! I have found the
script they use to do the checking:
https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
and I think we could probably implement something similar. I agree
that our build machines are much busier so I don't think we can use it
as a hard requirement for commit. The biggest open question in my mind
is how do they query JIRA for patches? Grab the correct patch off of
the JIRA? Schedule the job to poll for these things?
Brock
On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
> Hi Brock,
> that is an excellent idea and I'm definitely in favour of that. In fact I wanted to propose something similar myself.
>
> However I did not quite finished my own investigation yet. It seems that this pre commit hooks are done on projects that were mostly donated by Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named nodes have comment "Build machine for Hadoop project builds", so I'm not sure that they are available for other projects as well.
>
> With removing those 9 hadoop notes, it's like 10 other nodes in total for entire Apache organization and I do know that sometimes the build queue might be quite huge. Having said that it might be good idea to ask Infra team if pre-commit hooks are available to all projects or only to projects that have "special hardware" available.
>
> Jarcec
>
> On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
>> Hi,
>>
>> I like how in the Hadoop project if you mark a patch "Patch Available"
>> jenkins applies the patch to trunk, runs the tests, and outputs the
>> result to the JIRA. Here is one example:
>>
>> http://s.apache.org/qz
>>
>> I think we should still run tests manually before commit. However,
>> this will often give a reviewer an indicator of whether the change
>> breaks a test without even having to download the code.
>>
>> Thoughts?
>>
>> Brock
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Brock Noland 2012-12-07, 17:14
-
Re: [DISCUSS] Test on Patch AvailableJarek Jarcec Cecho 2012-12-07, 23:20
Btw I just found very interesting draft that is explaining quite huge portions of the design:
http://wiki.apache.org/general/PreCommitBuilds
Jarcec
On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
> Hi,
>
> Great to hear you were thinking the same thing! I have found the
> script they use to do the checking:
>
> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
>
> and I think we could probably implement something similar. I agree
> that our build machines are much busier so I don't think we can use it
> as a hard requirement for commit. The biggest open question in my mind
> is how do they query JIRA for patches? Grab the correct patch off of
> the JIRA? Schedule the job to poll for these things?
>
> Brock
>
> On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
> > Hi Brock,
> > that is an excellent idea and I'm definitely in favour of that. In fact I wanted to propose something similar myself.
> >
> > However I did not quite finished my own investigation yet. It seems that this pre commit hooks are done on projects that were mostly donated by Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named nodes have comment "Build machine for Hadoop project builds", so I'm not sure that they are available for other projects as well.
> >
> > With removing those 9 hadoop notes, it's like 10 other nodes in total for entire Apache organization and I do know that sometimes the build queue might be quite huge. Having said that it might be good idea to ask Infra team if pre-commit hooks are available to all projects or only to projects that have "special hardware" available.
> >
> > Jarcec
> >
> > On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
> >> Hi,
> >>
> >> I like how in the Hadoop project if you mark a patch "Patch Available"
> >> jenkins applies the patch to trunk, runs the tests, and outputs the
> >> result to the JIRA. Here is one example:
> >>
> >> http://s.apache.org/qz
> >>
> >> I think we should still run tests manually before commit. However,
> >> this will often give a reviewer an indicator of whether the change
> >> breaks a test without even having to download the code.
> >>
> >> Thoughts?
> >>
> >> Brock
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
http://wiki.apache.org/general/PreCommitBuilds
Jarcec
On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
> Hi,
>
> Great to hear you were thinking the same thing! I have found the
> script they use to do the checking:
>
> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
>
> and I think we could probably implement something similar. I agree
> that our build machines are much busier so I don't think we can use it
> as a hard requirement for commit. The biggest open question in my mind
> is how do they query JIRA for patches? Grab the correct patch off of
> the JIRA? Schedule the job to poll for these things?
>
> Brock
>
> On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
> > Hi Brock,
> > that is an excellent idea and I'm definitely in favour of that. In fact I wanted to propose something similar myself.
> >
> > However I did not quite finished my own investigation yet. It seems that this pre commit hooks are done on projects that were mostly donated by Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named nodes have comment "Build machine for Hadoop project builds", so I'm not sure that they are available for other projects as well.
> >
> > With removing those 9 hadoop notes, it's like 10 other nodes in total for entire Apache organization and I do know that sometimes the build queue might be quite huge. Having said that it might be good idea to ask Infra team if pre-commit hooks are available to all projects or only to projects that have "special hardware" available.
> >
> > Jarcec
> >
> > On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
> >> Hi,
> >>
> >> I like how in the Hadoop project if you mark a patch "Patch Available"
> >> jenkins applies the patch to trunk, runs the tests, and outputs the
> >> result to the JIRA. Here is one example:
> >>
> >> http://s.apache.org/qz
> >>
> >> I think we should still run tests manually before commit. However,
> >> this will often give a reviewer an indicator of whether the change
> >> breaks a test without even having to download the code.
> >>
> >> Thoughts?
> >>
> >> Brock
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Jarek Jarcec Cecho 2012-12-07, 23:20
-
Re: [DISCUSS] Test on Patch AvailableMike Percy 2012-12-08, 08:39
I really like the idea of doing this! +1 from me if we can swing it.
Regards,
Mike
On Fri, Dec 7, 2012 at 3:20 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
> Btw I just found very interesting draft that is explaining quite huge
> portions of the design:
>
> http://wiki.apache.org/general/PreCommitBuilds
>
> Jarcec
>
> On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
> > Hi,
> >
> > Great to hear you were thinking the same thing! I have found the
> > script they use to do the checking:
> >
> >
> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
> >
> > and I think we could probably implement something similar. I agree
> > that our build machines are much busier so I don't think we can use it
> > as a hard requirement for commit. The biggest open question in my mind
> > is how do they query JIRA for patches? Grab the correct patch off of
> > the JIRA? Schedule the job to poll for these things?
> >
> > Brock
> >
> > On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>
> wrote:
> > > Hi Brock,
> > > that is an excellent idea and I'm definitely in favour of that. In
> fact I wanted to propose something similar myself.
> > >
> > > However I did not quite finished my own investigation yet. It seems
> that this pre commit hooks are done on projects that were mostly donated by
> Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named
> nodes have comment "Build machine for Hadoop project builds", so I'm not
> sure that they are available for other projects as well.
> > >
> > > With removing those 9 hadoop notes, it's like 10 other nodes in total
> for entire Apache organization and I do know that sometimes the build queue
> might be quite huge. Having said that it might be good idea to ask Infra
> team if pre-commit hooks are available to all projects or only to projects
> that have "special hardware" available.
> > >
> > > Jarcec
> > >
> > > On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
> > >> Hi,
> > >>
> > >> I like how in the Hadoop project if you mark a patch "Patch Available"
> > >> jenkins applies the patch to trunk, runs the tests, and outputs the
> > >> result to the JIRA. Here is one example:
> > >>
> > >> http://s.apache.org/qz
> > >>
> > >> I think we should still run tests manually before commit. However,
> > >> this will often give a reviewer an indicator of whether the change
> > >> breaks a test without even having to download the code.
> > >>
> > >> Thoughts?
> > >>
> > >> Brock
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
Regards,
Mike
On Fri, Dec 7, 2012 at 3:20 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
> Btw I just found very interesting draft that is explaining quite huge
> portions of the design:
>
> http://wiki.apache.org/general/PreCommitBuilds
>
> Jarcec
>
> On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
> > Hi,
> >
> > Great to hear you were thinking the same thing! I have found the
> > script they use to do the checking:
> >
> >
> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
> >
> > and I think we could probably implement something similar. I agree
> > that our build machines are much busier so I don't think we can use it
> > as a hard requirement for commit. The biggest open question in my mind
> > is how do they query JIRA for patches? Grab the correct patch off of
> > the JIRA? Schedule the job to poll for these things?
> >
> > Brock
> >
> > On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>
> wrote:
> > > Hi Brock,
> > > that is an excellent idea and I'm definitely in favour of that. In
> fact I wanted to propose something similar myself.
> > >
> > > However I did not quite finished my own investigation yet. It seems
> that this pre commit hooks are done on projects that were mostly donated by
> Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named
> nodes have comment "Build machine for Hadoop project builds", so I'm not
> sure that they are available for other projects as well.
> > >
> > > With removing those 9 hadoop notes, it's like 10 other nodes in total
> for entire Apache organization and I do know that sometimes the build queue
> might be quite huge. Having said that it might be good idea to ask Infra
> team if pre-commit hooks are available to all projects or only to projects
> that have "special hardware" available.
> > >
> > > Jarcec
> > >
> > > On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
> > >> Hi,
> > >>
> > >> I like how in the Hadoop project if you mark a patch "Patch Available"
> > >> jenkins applies the patch to trunk, runs the tests, and outputs the
> > >> result to the JIRA. Here is one example:
> > >>
> > >> http://s.apache.org/qz
> > >>
> > >> I think we should still run tests manually before commit. However,
> > >> this will often give a reviewer an indicator of whether the change
> > >> breaks a test without even having to download the code.
> > >>
> > >> Thoughts?
> > >>
> > >> Brock
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
+
Mike Percy 2012-12-08, 08:39
-
Re: [DISCUSS] Test on Patch AvailableBrock Noland 2012-12-20, 06:38
Just an update here, the last two comments on this JIRA
https://issues.apache.org/jira/browse/FLUME-1787 are the messages on
failure and success. At present the only check it does is apply the
patch and run the tests. We can add more checks over time. I ended up
not using the Hadoop script because it is one ugly beast of a shell
script. The patches on that jira are only for testing this script.
I'll be submitting a real patch soon.
Brock
On Sat, Dec 8, 2012 at 2:39 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
> I really like the idea of doing this! +1 from me if we can swing it.
>
> Regards,
> Mike
>
>
> On Fri, Dec 7, 2012 at 3:20 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
>> Btw I just found very interesting draft that is explaining quite huge
>> portions of the design:
>>
>> http://wiki.apache.org/general/PreCommitBuilds
>>
>> Jarcec
>>
>> On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
>> > Hi,
>> >
>> > Great to hear you were thinking the same thing! I have found the
>> > script they use to do the checking:
>> >
>> >
>> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
>> >
>> > and I think we could probably implement something similar. I agree
>> > that our build machines are much busier so I don't think we can use it
>> > as a hard requirement for commit. The biggest open question in my mind
>> > is how do they query JIRA for patches? Grab the correct patch off of
>> > the JIRA? Schedule the job to poll for these things?
>> >
>> > Brock
>> >
>> > On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>
>> wrote:
>> > > Hi Brock,
>> > > that is an excellent idea and I'm definitely in favour of that. In
>> fact I wanted to propose something similar myself.
>> > >
>> > > However I did not quite finished my own investigation yet. It seems
>> that this pre commit hooks are done on projects that were mostly donated by
>> Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named
>> nodes have comment "Build machine for Hadoop project builds", so I'm not
>> sure that they are available for other projects as well.
>> > >
>> > > With removing those 9 hadoop notes, it's like 10 other nodes in total
>> for entire Apache organization and I do know that sometimes the build queue
>> might be quite huge. Having said that it might be good idea to ask Infra
>> team if pre-commit hooks are available to all projects or only to projects
>> that have "special hardware" available.
>> > >
>> > > Jarcec
>> > >
>> > > On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
>> > >> Hi,
>> > >>
>> > >> I like how in the Hadoop project if you mark a patch "Patch Available"
>> > >> jenkins applies the patch to trunk, runs the tests, and outputs the
>> > >> result to the JIRA. Here is one example:
>> > >>
>> > >> http://s.apache.org/qz
>> > >>
>> > >> I think we should still run tests manually before commit. However,
>> > >> this will often give a reviewer an indicator of whether the change
>> > >> breaks a test without even having to download the code.
>> > >>
>> > >> Thoughts?
>> > >>
>> > >> Brock
>> >
>> >
>> >
>> > --
>> > Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
https://issues.apache.org/jira/browse/FLUME-1787 are the messages on
failure and success. At present the only check it does is apply the
patch and run the tests. We can add more checks over time. I ended up
not using the Hadoop script because it is one ugly beast of a shell
script. The patches on that jira are only for testing this script.
I'll be submitting a real patch soon.
Brock
On Sat, Dec 8, 2012 at 2:39 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
> I really like the idea of doing this! +1 from me if we can swing it.
>
> Regards,
> Mike
>
>
> On Fri, Dec 7, 2012 at 3:20 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
>> Btw I just found very interesting draft that is explaining quite huge
>> portions of the design:
>>
>> http://wiki.apache.org/general/PreCommitBuilds
>>
>> Jarcec
>>
>> On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
>> > Hi,
>> >
>> > Great to hear you were thinking the same thing! I have found the
>> > script they use to do the checking:
>> >
>> >
>> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
>> >
>> > and I think we could probably implement something similar. I agree
>> > that our build machines are much busier so I don't think we can use it
>> > as a hard requirement for commit. The biggest open question in my mind
>> > is how do they query JIRA for patches? Grab the correct patch off of
>> > the JIRA? Schedule the job to poll for these things?
>> >
>> > Brock
>> >
>> > On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>
>> wrote:
>> > > Hi Brock,
>> > > that is an excellent idea and I'm definitely in favour of that. In
>> fact I wanted to propose something similar myself.
>> > >
>> > > However I did not quite finished my own investigation yet. It seems
>> that this pre commit hooks are done on projects that were mostly donated by
>> Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named
>> nodes have comment "Build machine for Hadoop project builds", so I'm not
>> sure that they are available for other projects as well.
>> > >
>> > > With removing those 9 hadoop notes, it's like 10 other nodes in total
>> for entire Apache organization and I do know that sometimes the build queue
>> might be quite huge. Having said that it might be good idea to ask Infra
>> team if pre-commit hooks are available to all projects or only to projects
>> that have "special hardware" available.
>> > >
>> > > Jarcec
>> > >
>> > > On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
>> > >> Hi,
>> > >>
>> > >> I like how in the Hadoop project if you mark a patch "Patch Available"
>> > >> jenkins applies the patch to trunk, runs the tests, and outputs the
>> > >> result to the JIRA. Here is one example:
>> > >>
>> > >> http://s.apache.org/qz
>> > >>
>> > >> I think we should still run tests manually before commit. However,
>> > >> this will often give a reviewer an indicator of whether the change
>> > >> breaks a test without even having to download the code.
>> > >>
>> > >> Thoughts?
>> > >>
>> > >> Brock
>> >
>> >
>> >
>> > --
>> > Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Brock Noland 2012-12-20, 06:38
-
Re: [DISCUSS] Test on Patch AvailableJarek Jarcec Cecho 2012-12-20, 15:04
Awesome, thank you very much Brock!
Jarcec
On Dec 19, 2012, at 10:38 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> Just an update here, the last two comments on this JIRA
> https://issues.apache.org/jira/browse/FLUME-1787 are the messages on
> failure and success. At present the only check it does is apply the
> patch and run the tests. We can add more checks over time. I ended up
> not using the Hadoop script because it is one ugly beast of a shell
> script. The patches on that jira are only for testing this script.
> I'll be submitting a real patch soon.
>
> Brock
>
> On Sat, Dec 8, 2012 at 2:39 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>> I really like the idea of doing this! +1 from me if we can swing it.
>>
>> Regards,
>> Mike
>>
>>
>> On Fri, Dec 7, 2012 at 3:20 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>>
>>> Btw I just found very interesting draft that is explaining quite huge
>>> portions of the design:
>>>
>>> http://wiki.apache.org/general/PreCommitBuilds
>>>
>>> Jarcec
>>>
>>> On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
>>>> Hi,
>>>>
>>>> Great to hear you were thinking the same thing! I have found the
>>>> script they use to do the checking:
>>>>
>>>>
>>> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
>>>>
>>>> and I think we could probably implement something similar. I agree
>>>> that our build machines are much busier so I don't think we can use it
>>>> as a hard requirement for commit. The biggest open question in my mind
>>>> is how do they query JIRA for patches? Grab the correct patch off of
>>>> the JIRA? Schedule the job to poll for these things?
>>>>
>>>> Brock
>>>>
>>>> On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>
>>> wrote:
>>>>> Hi Brock,
>>>>> that is an excellent idea and I'm definitely in favour of that. In
>>> fact I wanted to propose something similar myself.
>>>>>
>>>>> However I did not quite finished my own investigation yet. It seems
>>> that this pre commit hooks are done on projects that were mostly donated by
>>> Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named
>>> nodes have comment "Build machine for Hadoop project builds", so I'm not
>>> sure that they are available for other projects as well.
>>>>>
>>>>> With removing those 9 hadoop notes, it's like 10 other nodes in total
>>> for entire Apache organization and I do know that sometimes the build queue
>>> might be quite huge. Having said that it might be good idea to ask Infra
>>> team if pre-commit hooks are available to all projects or only to projects
>>> that have "special hardware" available.
>>>>>
>>>>> Jarcec
>>>>>
>>>>> On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I like how in the Hadoop project if you mark a patch "Patch Available"
>>>>>> jenkins applies the patch to trunk, runs the tests, and outputs the
>>>>>> result to the JIRA. Here is one example:
>>>>>>
>>>>>> http://s.apache.org/qz
>>>>>>
>>>>>> I think we should still run tests manually before commit. However,
>>>>>> this will often give a reviewer an indicator of whether the change
>>>>>> breaks a test without even having to download the code.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Brock
>>>>
>>>>
>>>>
>>>> --
>>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Jarcec
On Dec 19, 2012, at 10:38 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> Just an update here, the last two comments on this JIRA
> https://issues.apache.org/jira/browse/FLUME-1787 are the messages on
> failure and success. At present the only check it does is apply the
> patch and run the tests. We can add more checks over time. I ended up
> not using the Hadoop script because it is one ugly beast of a shell
> script. The patches on that jira are only for testing this script.
> I'll be submitting a real patch soon.
>
> Brock
>
> On Sat, Dec 8, 2012 at 2:39 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>> I really like the idea of doing this! +1 from me if we can swing it.
>>
>> Regards,
>> Mike
>>
>>
>> On Fri, Dec 7, 2012 at 3:20 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>>
>>> Btw I just found very interesting draft that is explaining quite huge
>>> portions of the design:
>>>
>>> http://wiki.apache.org/general/PreCommitBuilds
>>>
>>> Jarcec
>>>
>>> On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
>>>> Hi,
>>>>
>>>> Great to hear you were thinking the same thing! I have found the
>>>> script they use to do the checking:
>>>>
>>>>
>>> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
>>>>
>>>> and I think we could probably implement something similar. I agree
>>>> that our build machines are much busier so I don't think we can use it
>>>> as a hard requirement for commit. The biggest open question in my mind
>>>> is how do they query JIRA for patches? Grab the correct patch off of
>>>> the JIRA? Schedule the job to poll for these things?
>>>>
>>>> Brock
>>>>
>>>> On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>
>>> wrote:
>>>>> Hi Brock,
>>>>> that is an excellent idea and I'm definitely in favour of that. In
>>> fact I wanted to propose something similar myself.
>>>>>
>>>>> However I did not quite finished my own investigation yet. It seems
>>> that this pre commit hooks are done on projects that were mostly donated by
>>> Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named
>>> nodes have comment "Build machine for Hadoop project builds", so I'm not
>>> sure that they are available for other projects as well.
>>>>>
>>>>> With removing those 9 hadoop notes, it's like 10 other nodes in total
>>> for entire Apache organization and I do know that sometimes the build queue
>>> might be quite huge. Having said that it might be good idea to ask Infra
>>> team if pre-commit hooks are available to all projects or only to projects
>>> that have "special hardware" available.
>>>>>
>>>>> Jarcec
>>>>>
>>>>> On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I like how in the Hadoop project if you mark a patch "Patch Available"
>>>>>> jenkins applies the patch to trunk, runs the tests, and outputs the
>>>>>> result to the JIRA. Here is one example:
>>>>>>
>>>>>> http://s.apache.org/qz
>>>>>>
>>>>>> I think we should still run tests manually before commit. However,
>>>>>> this will often give a reviewer an indicator of whether the change
>>>>>> breaks a test without even having to download the code.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Brock
>>>>
>>>>
>>>>
>>>> --
>>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Jarek Jarcec Cecho 2012-12-20, 15:04
-
Re: [DISCUSS] Test on Patch AvailableBrock Noland 2012-12-21, 00:25
Alright, I have the patch up here:
https://issues.apache.org/jira/browse/FLUME-1787 Once it's committed I
can do the final testing and modifications to our pre-commit build
(https://builds.apache.org/job/PreCommit-FLUME-Build/) and we can turn
this sucker on.
Also, once we have the kinks worked out devs could use this script, I
have been using it to test patches this week.
Brock
On Thu, Dec 20, 2012 at 9:04 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
> Awesome, thank you very much Brock!
>
> Jarcec
>
> On Dec 19, 2012, at 10:38 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> Just an update here, the last two comments on this JIRA
>> https://issues.apache.org/jira/browse/FLUME-1787 are the messages on
>> failure and success. At present the only check it does is apply the
>> patch and run the tests. We can add more checks over time. I ended up
>> not using the Hadoop script because it is one ugly beast of a shell
>> script. The patches on that jira are only for testing this script.
>> I'll be submitting a real patch soon.
>>
>> Brock
>>
>> On Sat, Dec 8, 2012 at 2:39 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>> I really like the idea of doing this! +1 from me if we can swing it.
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Fri, Dec 7, 2012 at 3:20 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>>>
>>>> Btw I just found very interesting draft that is explaining quite huge
>>>> portions of the design:
>>>>
>>>> http://wiki.apache.org/general/PreCommitBuilds
>>>>
>>>> Jarcec
>>>>
>>>> On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
>>>>> Hi,
>>>>>
>>>>> Great to hear you were thinking the same thing! I have found the
>>>>> script they use to do the checking:
>>>>>
>>>>>
>>>> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
>>>>>
>>>>> and I think we could probably implement something similar. I agree
>>>>> that our build machines are much busier so I don't think we can use it
>>>>> as a hard requirement for commit. The biggest open question in my mind
>>>>> is how do they query JIRA for patches? Grab the correct patch off of
>>>>> the JIRA? Schedule the job to poll for these things?
>>>>>
>>>>> Brock
>>>>>
>>>>> On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>> Hi Brock,
>>>>>> that is an excellent idea and I'm definitely in favour of that. In
>>>> fact I wanted to propose something similar myself.
>>>>>>
>>>>>> However I did not quite finished my own investigation yet. It seems
>>>> that this pre commit hooks are done on projects that were mostly donated by
>>>> Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named
>>>> nodes have comment "Build machine for Hadoop project builds", so I'm not
>>>> sure that they are available for other projects as well.
>>>>>>
>>>>>> With removing those 9 hadoop notes, it's like 10 other nodes in total
>>>> for entire Apache organization and I do know that sometimes the build queue
>>>> might be quite huge. Having said that it might be good idea to ask Infra
>>>> team if pre-commit hooks are available to all projects or only to projects
>>>> that have "special hardware" available.
>>>>>>
>>>>>> Jarcec
>>>>>>
>>>>>> On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I like how in the Hadoop project if you mark a patch "Patch Available"
>>>>>>> jenkins applies the patch to trunk, runs the tests, and outputs the
>>>>>>> result to the JIRA. Here is one example:
>>>>>>>
>>>>>>> http://s.apache.org/qz
>>>>>>>
>>>>>>> I think we should still run tests manually before commit. However,
>>>>>>> this will often give a reviewer an indicator of whether the change
>>>>>>> breaks a test without even having to download the code.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Brock
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Apache MRUnit - Unit testing MapReduce -
>>>> http://incubator.apache.org/mrunit/
>>>>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
https://issues.apache.org/jira/browse/FLUME-1787 Once it's committed I
can do the final testing and modifications to our pre-commit build
(https://builds.apache.org/job/PreCommit-FLUME-Build/) and we can turn
this sucker on.
Also, once we have the kinks worked out devs could use this script, I
have been using it to test patches this week.
Brock
On Thu, Dec 20, 2012 at 9:04 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
> Awesome, thank you very much Brock!
>
> Jarcec
>
> On Dec 19, 2012, at 10:38 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> Just an update here, the last two comments on this JIRA
>> https://issues.apache.org/jira/browse/FLUME-1787 are the messages on
>> failure and success. At present the only check it does is apply the
>> patch and run the tests. We can add more checks over time. I ended up
>> not using the Hadoop script because it is one ugly beast of a shell
>> script. The patches on that jira are only for testing this script.
>> I'll be submitting a real patch soon.
>>
>> Brock
>>
>> On Sat, Dec 8, 2012 at 2:39 AM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>> I really like the idea of doing this! +1 from me if we can swing it.
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Fri, Dec 7, 2012 at 3:20 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>>>
>>>> Btw I just found very interesting draft that is explaining quite huge
>>>> portions of the design:
>>>>
>>>> http://wiki.apache.org/general/PreCommitBuilds
>>>>
>>>> Jarcec
>>>>
>>>> On Fri, Dec 07, 2012 at 11:14:13AM -0600, Brock Noland wrote:
>>>>> Hi,
>>>>>
>>>>> Great to hear you were thinking the same thing! I have found the
>>>>> script they use to do the checking:
>>>>>
>>>>>
>>>> https://github.com/apache/hadoop-common/blob/trunk/dev-support/test-patch.sh
>>>>>
>>>>> and I think we could probably implement something similar. I agree
>>>>> that our build machines are much busier so I don't think we can use it
>>>>> as a hard requirement for commit. The biggest open question in my mind
>>>>> is how do they query JIRA for patches? Grab the correct patch off of
>>>>> the JIRA? Schedule the job to poll for these things?
>>>>>
>>>>> Brock
>>>>>
>>>>> On Fri, Dec 7, 2012 at 10:26 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>> Hi Brock,
>>>>>> that is an excellent idea and I'm definitely in favour of that. In
>>>> fact I wanted to propose something similar myself.
>>>>>>
>>>>>> However I did not quite finished my own investigation yet. It seems
>>>> that this pre commit hooks are done on projects that were mostly donated by
>>>> Yahoo! (Hadoop, Oozie) on Jenkins nodes named "hadoopX". The hadoop named
>>>> nodes have comment "Build machine for Hadoop project builds", so I'm not
>>>> sure that they are available for other projects as well.
>>>>>>
>>>>>> With removing those 9 hadoop notes, it's like 10 other nodes in total
>>>> for entire Apache organization and I do know that sometimes the build queue
>>>> might be quite huge. Having said that it might be good idea to ask Infra
>>>> team if pre-commit hooks are available to all projects or only to projects
>>>> that have "special hardware" available.
>>>>>>
>>>>>> Jarcec
>>>>>>
>>>>>> On Fri, Dec 07, 2012 at 10:11:53AM -0600, Brock Noland wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I like how in the Hadoop project if you mark a patch "Patch Available"
>>>>>>> jenkins applies the patch to trunk, runs the tests, and outputs the
>>>>>>> result to the JIRA. Here is one example:
>>>>>>>
>>>>>>> http://s.apache.org/qz
>>>>>>>
>>>>>>> I think we should still run tests manually before commit. However,
>>>>>>> this will often give a reviewer an indicator of whether the change
>>>>>>> breaks a test without even having to download the code.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Brock
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Apache MRUnit - Unit testing MapReduce -
>>>> http://incubator.apache.org/mrunit/
>>>>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Brock Noland 2012-12-21, 00:25
