Hi all,

A while ago I created a feature for Hadoop and submitted this to be
included (HADOOP-7076) .
Around the same time the MRv2 started happening and the entire source tree
was restructured.

At this moment I'm prepared to change the patch I created earlier so I can
submit it again for your consideration.

Caused by the email about the new branches (branch-1 and branch-1.0) I'm a
bit puzzled at this moment where to start.

I see the mentioned branches and the trunk at probable starting points.

As far as I understand the repository structure the branch-1 is the basis
for the "old style" Hadoop and the trunk is the basis for the "yarn" Hadoop.

For which branch of the source tree should I make my changes so you guys
will reevaluate it for inclusion?

Thanks.

--
Best regards / Met vriendelijke groeten,

Niels Basjes

Niels,

I think that the branch you put it on depends mostly on where you and others what to see this feature (splittable Gzip) go in.  At a minimum you should target trunk.  If you want to see it go into 1.* then you probably also want to port it to that line (branch-1).  Once they are in porting it to other branches should be fairly simple and really depends on where you and others want to use your feature.

--Bobby Evans

On 11/30/11 10:23 AM, "Niels Basjes" <[EMAIL PROTECTED]> wrote:

Hi all,

A while ago I created a feature for Hadoop and submitted this to be
included (HADOOP-7076) .
Around the same time the MRv2 started happening and the entire source tree
was restructured.

At this moment I'm prepared to change the patch I created earlier so I can
submit it again for your consideration.

Caused by the email about the new branches (branch-1 and branch-1.0) I'm a
bit puzzled at this moment where to start.

I see the mentioned branches and the trunk at probable starting points.

As far as I understand the repository structure the branch-1 is the basis
for the "old style" Hadoop and the trunk is the basis for the "yarn" Hadoop.

For which branch of the source tree should I make my changes so you guys
will reevaluate it for inclusion?

Thanks.

--
Best regards / Met vriendelijke groeten,

Niels Basjes

Hey Niels,

Thanks for contributing.  The nest place to contribute new features is
to trunk. It's currently an easy merge from trunk to branch 23 to get
it in a 23.x release (you can set the jira's target version to 23.1 to
indicate this).

Your patch based on the old structure would be useful for backporting
this feature from trunk to a release with the old structure (eg 1.x,
0.22). To request inclusion in a 1.x release set the target version to
1.1.0 (and generate a patch against branch-1). To request inclusion in
0.22 set target version to 0.22.0 (and generate a patch against
branch-0.22).

Thanks,
Eli

On Wed, Nov 30, 2011 at 8:23 AM, Niels Basjes <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> A while ago I created a feature for Hadoop and submitted this to be
> included (HADOOP-7076) .
> Around the same time the MRv2 started happening and the entire source tree
> was restructured.
>
> At this moment I'm prepared to change the patch I created earlier so I can
> submit it again for your consideration.
>
> Caused by the email about the new branches (branch-1 and branch-1.0) I'm a
> bit puzzled at this moment where to start.
>
> I see the mentioned branches and the trunk at probable starting points.
>
> As far as I understand the repository structure the branch-1 is the basis
> for the "old style" Hadoop and the trunk is the basis for the "yarn" Hadoop.
>
> For which branch of the source tree should I make my changes so you guys
> will reevaluate it for inclusion?
>
> Thanks.
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes

Thanks,

I'll get busy creating a new patch over the next few days.

Niels Basjes

On Wed, Nov 30, 2011 at 18:51, Eli Collins <[EMAIL PROTECTED]> wrote:

> Hey Niels,
>
> Thanks for contributing.  The nest place to contribute new features is
> to trunk. It's currently an easy merge from trunk to branch 23 to get
> it in a 23.x release (you can set the jira's target version to 23.1 to
> indicate this).
>
> Your patch based on the old structure would be useful for backporting
> this feature from trunk to a release with the old structure (eg 1.x,
> 0.22). To request inclusion in a 1.x release set the target version to
> 1.1.0 (and generate a patch against branch-1). To request inclusion in
> 0.22 set target version to 0.22.0 (and generate a patch against
> branch-0.22).
>
> Thanks,
> Eli
>
> On Wed, Nov 30, 2011 at 8:23 AM, Niels Basjes <[EMAIL PROTECTED]> wrote:
> > Hi all,
> >
> > A while ago I created a feature for Hadoop and submitted this to be
> > included (HADOOP-7076) .
> > Around the same time the MRv2 started happening and the entire source
> tree
> > was restructured.
> >
> > At this moment I'm prepared to change the patch I created earlier so I
> can
> > submit it again for your consideration.
> >
> > Caused by the email about the new branches (branch-1 and branch-1.0) I'm
> a
> > bit puzzled at this moment where to start.
> >
> > I see the mentioned branches and the trunk at probable starting points.
> >
> > As far as I understand the repository structure the branch-1 is the basis
> > for the "old style" Hadoop and the trunk is the basis for the "yarn"
> Hadoop.
> >
> > For which branch of the source tree should I make my changes so you guys
> > will reevaluate it for inclusion?
> >
> > Thanks.
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>

--
Best regards / Met vriendelijke groeten,

Niels Basjes

Hi,

On Wed, Nov 30, 2011 at 18:51, Eli Collins <[EMAIL PROTECTED]> wrote:

> Thanks for contributing.  The nest place to contribute new features is
>> to trunk. It's currently an easy merge from trunk to branch 23 to get
>> it in a 23.x release (you can set the jira's target version to 23.1 to
>> indicate this).
>>
>
I've just uploaded the new patch created against the trunk and set the
target for 0.23.1 as you indicated.

https://issues.apache.org/jira/browse/HADOOP-7076

Jenkins has just completed.
Although it passed everything else it was '-1' because of 9 javadoc
warnings that do not seem related to my patch.

https://builds.apache.org/job/PreCommit-HADOOP-Build/432/artifact/trunk/hadoop-common-project/patchprocess/patchJavadocWarnings.txt

Your patch based on the old structure would be useful for backporting
>> this feature from trunk to a release with the old structure (eg 1.x,
>> 0.22). To request inclusion in a 1.x release set the target version to
>> 1.1.0 (and generate a patch against branch-1). To request inclusion in
>> 0.22 set target version to 0.22.0 (and generate a patch against
>> branch-0.22).
>>
>
Do I simply make separate Jira (related) issues for these backports?

--
Best regards / Met vriendelijke groeten,

Niels Basjes

On Sun, Dec 4, 2011 at 3:30 PM, Niels Basjes <[EMAIL PROTECTED]> wrote:
> Hi,
>
> On Wed, Nov 30, 2011 at 18:51, Eli Collins <[EMAIL PROTECTED]> wrote:
>
>> Thanks for contributing.  The nest place to contribute new features is
>>> to trunk. It's currently an easy merge from trunk to branch 23 to get
>>> it in a 23.x release (you can set the jira's target version to 23.1 to
>>> indicate this).
>>>
>>
> I've just uploaded the new patch created against the trunk and set the
> target for 0.23.1 as you indicated.
>
> https://issues.apache.org/jira/browse/HADOOP-7076
>
> Jenkins has just completed.
> Although it passed everything else it was '-1' because of 9 javadoc
> warnings that do not seem related to my patch.
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/432/artifact/trunk/hadoop-common-project/patchprocess/patchJavadocWarnings.txt
>

Yea, these are not due to your patch. I'll bump the javadoc warnings
in test-patch.properties.

> Your patch based on the old structure would be useful for backporting
>>> this feature from trunk to a release with the old structure (eg 1.x,
>>> 0.22). To request inclusion in a 1.x release set the target version to
>>> 1.1.0 (and generate a patch against branch-1). To request inclusion in
>>> 0.22 set target version to 0.22.0 (and generate a patch against
>>> branch-0.22).
>>>
>>
> Do I simply make separate Jira (related) issues for these backports?
>

Nope, just set the "target version" field to the appropriate version
and uploaded a patch, eg hadoop-7076-branch-1.patch.

Thanks,
Eli
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes

Hi,

> https://issues.apache.org/jira/browse/HADOOP-7076
> >
> > Jenkins has just completed.
> > Although it passed everything else it was '-1' because of 9 javadoc
> > warnings that do not seem related to my patch.
> >
> >
> https://builds.apache.org/job/PreCommit-HADOOP-Build/432/artifact/trunk/hadoop-common-project/patchprocess/patchJavadocWarnings.txt
>
> Yea, these are not due to your patch. I'll bump the javadoc warnings
> in test-patch.properties.
Thanks.

> >>> Your patch based on the old structure would be useful for backporting
> >>> this feature from trunk to a release with the old structure (eg 1.x,
> >>> 0.22). To request inclusion in a 1.x release set the target version to
> >>> 1.1.0 (and generate a patch against branch-1). To request inclusion in
> >>> 0.22 set target version to 0.22.0 (and generate a patch against
> >>> branch-0.22).
>
> > Do I simply make separate Jira (related) issues for these backports?
>
> Nope, just set the "target version" field to the appropriate version
> and uploaded a patch, eg hadoop-7076-branch-1.patch.
>

So that I understand correctly:
- I add the targets to the Jira issue for each branch specific patch.
- I create a new patch file for each version I want the feature to appear
in and attach these to the issue.
- I name these patches something like <issue id>-<date>-<branchid>.patch so
that the committer can clearly see what it was intended for.

Do I have to do something to ensure Jenkins will accept this all correctly?
Perhaps in naming convention? Or in the timing between uploading the
various patches?

--
Best regards / Met vriendelijke groeten,

Niels Basjes

On Mon, Dec 5, 2011 at 6:09 AM, Niels Basjes <[EMAIL PROTECTED]> wrote:
> Hi,
>
>> https://issues.apache.org/jira/browse/HADOOP-7076
>> >
>> > Jenkins has just completed.
>> > Although it passed everything else it was '-1' because of 9 javadoc
>> > warnings that do not seem related to my patch.
>> >
>> >
>> https://builds.apache.org/job/PreCommit-HADOOP-Build/432/artifact/trunk/hadoop-common-project/patchprocess/patchJavadocWarnings.txt
>>
>> Yea, these are not due to your patch. I'll bump the javadoc warnings
>> in test-patch.properties.
>
>
> Thanks.
>
>
>
>> >>> Your patch based on the old structure would be useful for backporting
>> >>> this feature from trunk to a release with the old structure (eg 1.x,
>> >>> 0.22). To request inclusion in a 1.x release set the target version to
>> >>> 1.1.0 (and generate a patch against branch-1). To request inclusion in
>> >>> 0.22 set target version to 0.22.0 (and generate a patch against
>> >>> branch-0.22).
>>
>> > Do I simply make separate Jira (related) issues for these backports?
>>
>> Nope, just set the "target version" field to the appropriate version
>> and uploaded a patch, eg hadoop-7076-branch-1.patch.
>>
>
> So that I understand correctly:
> - I add the targets to the Jira issue for each branch specific patch.
> - I create a new patch file for each version I want the feature to appear
> in and attach these to the issue.
> - I name these patches something like <issue id>-<date>-<branchid>.patch so
> that the committer can clearly see what it was intended for.
>

Yup. See the steps listed in the "Generating a patch" section of this page:
http://wiki.apache.org/hadoop/HowToContribute
> Do I have to do something to ensure Jenkins will accept this all correctly?
> Perhaps in naming convention? Or in the timing between uploading the
> various patches?

Unfortunately jenkins doesn't currently run tests against non-trunk
trees.  For these branches you need to run test-patch (covered in the
above page) and the tests yourself.

Thanks,
Eli
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes

Hi,

On Mon, Dec 5, 2011 at 18:54, Eli Collins <[EMAIL PROTECTED]> wrote:

> >> https://issues.apache.org/jira/browse/HADOOP-7076
>
> >> >>> Your patch based on the old structure would be useful for
> backporting
> >> >>> this feature from trunk to a release with the old structure (eg 1.x,
> >> >>> 0.22). To request inclusion in a 1.x release set the target version
> to
> >> >>> 1.1.0 (and generate a patch against branch-1). To request inclusion
> in
> >> >>> 0.22 set target version to 0.22.0 (and generate a patch against
> >> >>> branch-0.22).
>

Turns out my changes are not trivial to backport to branch-1 because the
SplittableCompressionCodec interface isn't there yet.
So I'm limiting to trunk (0.23.1) and branch-0.22 (0.22.0).

...

>Unfortunately jenkins doesn't currently run tests against non-trunk
>trees.  For these branches you need to run test-patch (covered in the
>above page) and the tests yourself.

Unfortunately the test-patch script mentioned on the site
("dev-support/test-patch.sh /path/to/my.patch") doesn't exist in the
branch-0.22. And I couldn't get the test-patch.sh that does exist to work.

So I did against a clean checkout:
  patch -p0 < HADOOP-7076-branch-0.22.patch
followed by
  ant clean test jar

Which succeeded.

--
Best regards / Met vriendelijke groeten,

Niels Basjes

I got this:

Hadoop QA commented on HADOOP-7076:
-1 overall.  Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12506182/HADOOP-7076-branch-0.22.patch
  against trunk revision .
...
-1 patch.  The patch command could not apply the patch.

Did I do something wrong in the patch I created for branch-0.22?
Or is HADOOP-7435 not yet operational?

Thanks.

Niels Basjes
On Tue, Dec 6, 2011 at 00:17, Niels Basjes <[EMAIL PROTECTED]> wrote:

> Hi,
>
> On Mon, Dec 5, 2011 at 18:54, Eli Collins <[EMAIL PROTECTED]> wrote:
>
>> >> https://issues.apache.org/jira/browse/HADOOP-7076
>>
>
>
>> >> >>> Your patch based on the old structure would be useful for
>> backporting
>> >> >>> this feature from trunk to a release with the old structure (eg
>> 1.x,
>> >> >>> 0.22). To request inclusion in a 1.x release set the target
>> version to
>> >> >>> 1.1.0 (and generate a patch against branch-1). To request
>> inclusion in
>> >> >>> 0.22 set target version to 0.22.0 (and generate a patch against
>> >> >>> branch-0.22).
>>
>
> Turns out my changes are not trivial to backport to branch-1 because the
> SplittableCompressionCodec interface isn't there yet.
> So I'm limiting to trunk (0.23.1) and branch-0.22 (0.22.0).
>
> ...
>
>
> >Unfortunately jenkins doesn't currently run tests against non-trunk
> >trees.  For these branches you need to run test-patch (covered in the
> >above page) and the tests yourself.
>
> Unfortunately the test-patch script mentioned on the site
> ("dev-support/test-patch.sh /path/to/my.patch") doesn't exist in the
> branch-0.22. And I couldn't get the test-patch.sh that does exist to work.
>
> So I did against a clean checkout:
>   patch -p0 < HADOOP-7076-branch-0.22.patch
> followed by
>   ant clean test jar
>
> Which succeeded.
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

--
Best regards / Met vriendelijke groeten,

Niels Basjes

On Tue, Dec 6, 2011 at 1:54 PM, Niels Basjes <[EMAIL PROTECTED]> wrote:
> I got this:
>
> Hadoop QA commented on HADOOP-7076:
> -1 overall.  Here are the results of testing the latest attachment
>
> http://issues.apache.org/jira/secure/attachment/12506182/HADOOP-7076-branch-0.22.patch
>  against trunk revision .
> ...
> -1 patch.  The patch command could not apply the patch.
>
> Did I do something wrong in the patch I created for branch-0.22?
> Or is HADOOP-7435 not yet operational?
>

HADOOP-7435 is not yet fixed, the auto qa bot only works on patches for trunk.