Hadoop, mail # general - Re: HADOOP-7106 (project unsplit) this weekend

HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-10, 18:20

Hi all,

Pending any unforeseen issues, I am planning on committing HADOOP-7106 this
weekend. I have the credentials from Jukka to take care of the git trees as
well, and have done a "practice" move several times on a local mirror of the
svn.

I'll send out an announcement of the exact time in advance of when I
actually do the commit.

Thanks
-Todd
--
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Eli Collins 2011-06-10, 18:34

Spectacular. Thanks Todd!

On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> Pending any unforeseen issues, I am planning on committing HADOOP-7106 this
> weekend. I have the credentials from Jukka to take care of the git trees as
> well, and have done a "practice" move several times on a local mirror of the
> svn.
>
> I'll send out an announcement of the exact time in advance of when I
> actually do the commit.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: HADOOP-7106 (project unsplit) this weekend

Owen O'Malley 2011-06-10, 18:40

On Jun 10, 2011, at 11:20 AM, Todd Lipcon wrote:

> Pending any unforeseen issues, I am planning on committing HADOOP-7106 this
> weekend.

Thanks Todd for all of your work getting the details worked out and tested.

-- Owen

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-12, 03:25

Hi all,

I'm figuring out one more small nit I noticed in my testing this evening.
Hopefully I will figure out what's going wrong and be ready to press the big
button tomorrow.

Assuming I don't have to "abort mission", my hope is to do this at around
3PM PST tomorrow (Sunday). I'll send out a message asking folks to please
hold commits to all branches while the move is in progress.

Thanks
-Todd

On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> Pending any unforeseen issues, I am planning on committing HADOOP-7106 this
> weekend. I have the credentials from Jukka to take care of the git trees as
> well, and have done a "practice" move several times on a local mirror of the
> svn.
>
> I'll send out an announcement of the exact time in advance of when I
> actually do the commit.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-12, 21:50

All of the nits I ran into should be resolved and we should be good to go. I
will start this in just about 10 minutes (3pm PST).

***Please hold all commits until further notice!*** I anticipate that this
should take under an hour, but if there are any bumps along the way it might
stretch into the evening. I'll send out an "all clear" email when things are
ready to go on the new layout.

I've disabled all of the Hudson builds for now and will be re-enabling them
one by one after reconfiguring their SVN URLs.

-Todd

On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I'm figuring out one more small nit I noticed in my testing this evening.
> Hopefully I will figure out what's going wrong and be ready to press the big
> button tomorrow.
>
> Assuming I don't have to "abort mission", my hope is to do this at around
> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to please
> hold commits to all branches while the move is in progress.
>
> Thanks
> -Todd
>
>
> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> Hi all,
>>
>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>> this weekend. I have the credentials from Jukka to take care of the git
>> trees as well, and have done a "practice" move several times on a local
>> mirror of the svn.
>>
>> I'll send out an announcement of the exact time in advance of when I
>> actually do the commit.
>>
>> Thanks
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-12, 23:38

OK, this seems to have succeeded without any big problems!

I've re-enabled the git mirrors and the hudson builds. Feel free to commit
to the new trees.

Here are some instructions for the migration:

=== SVN users ==
Next time you "svn up" in your "common" working directory you'll end up
seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirectory.
This is probably the easiest place from which to work, now. The URLs for the
combined SVN trees are:

trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
branch-0.22:
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
branch-0.21:
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
yahoo-merge:
http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
  (this one has the yahoo-merge branches from common, hdfs, and mapred)
MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)

The same kind of thing happened for HDFS-1073 and branch-0.21-old.
Pre-project-split branches like branch-0.20 should have remained untouched.

You can proceed to delete your checkouts of the individual mapred and hdfs
trees, since they exist within the combined trees above. If for some reason
you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
its new location, you can use the following incantation:
svn sw $(svn info | grep URL | awk '{print $2}' | sed
's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')

=== Git Users ==The git mirrors of the above 7 branches should now have a set of 4 commits
near the top that look like this:

Merge: 928d485 cd66945 77f628f
Author: Todd Lipcon <[EMAIL PROTECTED]>
Date: Sun Jun 12 22:53:28 2011 +0000

   HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
single tree (project unsplit)

   git-svn-id:
https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68

commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
Author: Todd Lipcon <[EMAIL PROTECTED]>
Date: Sun Jun 12 22:53:27 2011 +0000

   Relocate mapreduce into mapreduce/

commit cd66945f62635f589ff93468e94c0039684a8b6d
Author: Todd Lipcon <[EMAIL PROTECTED]>
Date: Sun Jun 12 22:53:26 2011 +0000

   Relocate hdfs into hdfs/

commit 928d485e2743115fe37f9d123ce9a635c5afb91a
Author: Todd Lipcon <[EMAIL PROTECTED]>
Date: Sun Jun 12 22:53:25 2011 +0000

   Relocate common into common/

The first of these 4 is a 3-parent "octopus" merge commit of the
pre-project-unsplit branches. In theory, git is smart enough to track
changes through this merge, so long as you pass the right flags (eg
--follow). For example:

todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline --abbrev-commit
--follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
-10
77f628f Relocate mapreduce into mapreduce/
90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
JobTrackerStatus.
ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
aid diagnosis of related issues. Contributed by Jonathan Eagles
32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
Contributed by Ari Rabkin.

If you want to be able to have git follow renames all the way through the
project split back to the beginning of time, put the following in
hadoop-common/.git/info/grafts:
5128a9a453d64bfe1ed978cf9ffed27985eeef36
6c16dc8cf2b28818c852e95302920a278d07ad0c
6a3ac690e493c7da45bbf2ae2054768c427fd0e1
6c16dc8cf2b28818c852e95302920a278d07ad0c
546d96754ffee3142bcbbf4563c624c053d0ed0d
6c16dc8cf2b28818c852e95302920a278d07ad0c

In terms of rebasing git branches, git is actually pretty smart. For
example, I have a local "HDFS-1073" branch in my hdfs repo. To transition it
to the new combined repo, I did the following:

# Add my project-split hdfs git repo as a remote:
git remote add splithdfs /home/todd/git/hadoop-hdfs/
git fetch splithdfs

# Checkout a branch in my combined repo
git checkout -b HDFS-1073 splithdfs/HDFS-1073

# Rebase it on the combined 1073 branch
git rebase origin/HDFS-1073

...and it actually applies my patches inside the appropriate subdirectory (I
was surprised and impressed by this!)
If the branch you're rebasing has added or moved files, it might not be
smart enough and you'll have to manually rename them in your branch inside
of the appropriate subtree.. but for simple patches this seems to work. For
less simple things, the best bet may be to use "git filter-branch" on the
patch series to relocate it inside a subdirectory, and then try to rebase.
Let me know if you need a hand with any git cleanup, happy to help.
== Outstanding issues =
The one outstanding issue I'm aware of is that the test-patch builds should
be smart enough to be able to deal with patches that are relative to the
combined root instead of the original project. Right now, if you export a
diff from git, it will include "hdfs/" or "mapreduce/" in the changed file
names, and the QA bot won't know how to apply it. The workaround for this is
to change directory into the relative subproject dir, and then pass
"--relative" to "git diff" or "git show", for example:

todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
diff --git CHANGES.txt CHANGES.txt
...
I imagine there are probably some other things that fell through the cracks.
Please get in touch if there's anything that seems amiss.

-Todd
On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Eli Collins 2011-06-13, 07:39

Nice work Todd. Thank you for making this happen!
On Sun, Jun 12, 2011 at 4:38 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> OK, this seems to have succeeded without any big problems!
>
> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
> to the new trees.
>
> Here are some instructions for the migration:
>
> === SVN users ==>
> Next time you "svn up" in your "common" working directory you'll end up
> seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirectory.
> This is probably the easiest place from which to work, now. The URLs for the
> combined SVN trees are:
>
> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> branch-0.22:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> branch-0.21:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> yahoo-merge:
> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
> (this one has the yahoo-merge branches from common, hdfs, and mapred)
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
> (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>
> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> Pre-project-split branches like branch-0.20 should have remained untouched.
>
> You can proceed to delete your checkouts of the individual mapred and hdfs
> trees, since they exist within the combined trees above. If for some reason
> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
> its new location, you can use the following incantation:
> svn sw $(svn info | grep URL | awk '{print $2}' | sed
> 's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')
>
> === Git Users ==> The git mirrors of the above 7 branches should now have a set of 4 commits
> near the top that look like this:
>
> Merge: 928d485 cd66945 77f628f
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:28 2011 +0000
>
> HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
> single tree (project unsplit)
>
> git-svn-id:
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68
>
> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:27 2011 +0000
>
> Relocate mapreduce into mapreduce/
>
> commit cd66945f62635f589ff93468e94c0039684a8b6d
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:26 2011 +0000
>
> Relocate hdfs into hdfs/
>
> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:25 2011 +0000
>
> Relocate common into common/
>
> The first of these 4 is a 3-parent "octopus" merge commit of the
> pre-project-unsplit branches. In theory, git is smart enough to track
> changes through this merge, so long as you pass the right flags (eg
> --follow). For example:
>
> todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline --abbrev-commit
> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
> -10
> 77f628f Relocate mapreduce into mapreduce/
> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
> JobTrackerStatus.
> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
> aid diagnosis of related issues. Contributed by Jonathan Eagles
> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
> Contributed by Ari Rabkin.
>
> If you want to be able to have git follow renames all the way through the
> project split back to the beginning of time, put the following in
> hadoop-common/.git/info/grafts:
> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 546d96754ffee3142bcbbf4563c624c053d0ed0d
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>
> In terms of rebasing git branches, git is actually pretty smart. For
> example, I have a local "HDFS-1073" branch in my hdfs repo. To transition it

Re: HADOOP-7106 (project unsplit) this weekend

Robert Evans 2011-06-13, 14:05

Could someone unlock some of these branches for anonymous read only checkout? At least with MR-279 I get a 403 forbidden error when I try to check out.

--Bobby

On 6/12/11 6:38 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:

OK, this seems to have succeeded without any big problems!

I've re-enabled the git mirrors and the hudson builds. Feel free to commit
to the new trees.

Here are some instructions for the migration:

=== SVN users ==
Next time you "svn up" in your "common" working directory you'll end up
seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirectory.
This is probably the easiest place from which to work, now. The URLs for the
combined SVN trees are:

trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
branch-0.22:
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
branch-0.21:
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
yahoo-merge:
http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
  (this one has the yahoo-merge branches from common, hdfs, and mapred)
MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)

The same kind of thing happened for HDFS-1073 and branch-0.21-old.
Pre-project-split branches like branch-0.20 should have remained untouched.

You can proceed to delete your checkouts of the individual mapred and hdfs
trees, since they exist within the combined trees above. If for some reason
you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
its new location, you can use the following incantation:
svn sw $(svn info | grep URL | awk '{print $2}' | sed
's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')

=== Git Users ==The git mirrors of the above 7 branches should now have a set of 4 commits
near the top that look like this:

Merge: 928d485 cd66945 77f628f
Author: Todd Lipcon <[EMAIL PROTECTED]>
Date: Sun Jun 12 22:53:28 2011 +0000

   HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
single tree (project unsplit)

   git-svn-id:
https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68

commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
Author: Todd Lipcon <[EMAIL PROTECTED]>
Date: Sun Jun 12 22:53:27 2011 +0000

   Relocate mapreduce into mapreduce/

commit cd66945f62635f589ff93468e94c0039684a8b6d
Author: Todd Lipcon <[EMAIL PROTECTED]>
Date: Sun Jun 12 22:53:26 2011 +0000

   Relocate hdfs into hdfs/

commit 928d485e2743115fe37f9d123ce9a635c5afb91a
Author: Todd Lipcon <[EMAIL PROTECTED]>
Date: Sun Jun 12 22:53:25 2011 +0000

   Relocate common into common/

The first of these 4 is a 3-parent "octopus" merge commit of the
pre-project-unsplit branches. In theory, git is smart enough to track
changes through this merge, so long as you pass the right flags (eg
--follow). For example:

todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline --abbrev-commit
--follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
-10
77f628f Relocate mapreduce into mapreduce/
90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
JobTrackerStatus.
ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
aid diagnosis of related issues. Contributed by Jonathan Eagles
32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
Contributed by Ari Rabkin.

If you want to be able to have git follow renames all the way through the
project split back to the beginning of time, put the following in
hadoop-common/.git/info/grafts:
5128a9a453d64bfe1ed978cf9ffed27985eeef36
6c16dc8cf2b28818c852e95302920a278d07ad0c
6a3ac690e493c7da45bbf2ae2054768c427fd0e1
6c16dc8cf2b28818c852e95302920a278d07ad0c
546d96754ffee3142bcbbf4563c624c053d0ed0d
6c16dc8cf2b28818c852e95302920a278d07ad0c

In terms of rebasing git branches, git is actually pretty smart. For
example, I have a local "HDFS-1073" branch in my hdfs repo. To transition it
to the new combined repo, I did the following:

# Add my project-split hdfs git repo as a remote:
git remote add splithdfs /home/todd/git/hadoop-hdfs/
git fetch splithdfs

# Checkout a branch in my combined repo
git checkout -b HDFS-1073 splithdfs/HDFS-1073

# Rebase it on the combined 1073 branch
git rebase origin/HDFS-1073

...and it actually applies my patches inside the appropriate subdirectory (I
was surprised and impressed by this!)
If the branch you're rebasing has added or moved files, it might not be
smart enough and you'll have to manually rename them in your branch inside
of the appropriate subtree.. but for simple patches this seems to work. For
less simple things, the best bet may be to use "git filter-branch" on the
patch series to relocate it inside a subdirectory, and then try to rebase.
Let me know if you need a hand with any git cleanup, happy to help.
== Outstanding issues =
The one outstanding issue I'm aware of is that the test-patch builds should
be smart enough to be able to deal with patches that are relative to the
combined root instead of the original project. Right now, if you export a
diff from git, it will include "hdfs/" or "mapreduce/" in the changed file
names, and the QA bot won't know how to apply it. The workaround for this is
to change directory into the relative subproject dir, and then pass
"--relative" to "git diff" or "git show", for example:

todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
diff --git CHANGES.txt CHANGES.txt
...
I imagine there are probably some other things that fell through the cracks.
Please get in touch if there's anything that seems amiss.

-Todd
On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-13, 14:52

Hey Robert,

It seems to be working for me... what URL are you trying to check out?
I moved aside my ~/.subversion dir, and then did:
$ svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/

Thanks
-Todd

On Mon, Jun 13, 2011 at 7:05 AM, Robert Evans <[EMAIL PROTECTED]> wrote:

> Could someone unlock some of these branches for anonymous read only
> checkout? At least with MR-279 I get a 403 forbidden error when I try to
> check out.
>
> --Bobby
>
> On 6/12/11 6:38 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:
>
> OK, this seems to have succeeded without any big problems!
>
> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
> to the new trees.
>
> Here are some instructions for the migration:
>
> === SVN users ==>
> Next time you "svn up" in your "common" working directory you'll end up
> seeing the combined tree - ie a mapreduce/, hdfs/, and common/
> subdirectory.
> This is probably the easiest place from which to work, now. The URLs for
> the
> combined SVN trees are:
>
> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> branch-0.22:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> branch-0.21:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> yahoo-merge:
> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
> (this one has the yahoo-merge branches from common, hdfs, and mapred)
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
> (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>
> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> Pre-project-split branches like branch-0.20 should have remained untouched.
>
> You can proceed to delete your checkouts of the individual mapred and hdfs
> trees, since they exist within the combined trees above. If for some reason
> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
> its new location, you can use the following incantation:
> svn sw $(svn info | grep URL | awk '{print $2}' | sed
> 's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')
>
> === Git Users ==> The git mirrors of the above 7 branches should now have a set of 4 commits
> near the top that look like this:
>
> Merge: 928d485 cd66945 77f628f
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:28 2011 +0000
>
> HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
> single tree (project unsplit)
>
> git-svn-id:
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-<https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68>
> 0310-9956-ffa450edef68
>
> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:27 2011 +0000
>
> Relocate mapreduce into mapreduce/
>
> commit cd66945f62635f589ff93468e94c0039684a8b6d
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:26 2011 +0000
>
> Relocate hdfs into hdfs/
>
> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:25 2011 +0000
>
> Relocate common into common/
>
> The first of these 4 is a 3-parent "octopus" merge commit of the
> pre-project-unsplit branches. In theory, git is smart enough to track
> changes through this merge, so long as you pass the right flags (eg
> --follow). For example:
>
> todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline
> --abbrev-commit
> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
> -10
> 77f628f Relocate mapreduce into mapreduce/
> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
> JobTrackerStatus.
> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
> aid diagnosis of related issues. Contributed by Jonathan Eagles
> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
> Contributed by Ari Rabkin.
>
> If you want to be able to have git follow renames all the way through the
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-13, 14:57

Hmm, as I got farther down my email this morning, I saw some complaints that
HBase's SVN was giving 403s as well, this morning. Perhaps there is some
ASF-wide issue going on, completely unrelated to the Hadoop changes over the
weekend?

-Todd

On Mon, Jun 13, 2011 at 7:52 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hey Robert,
>
> It seems to be working for me... what URL are you trying to check out?
> I moved aside my ~/.subversion dir, and then did:
> $ svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/
>
> Thanks
> -Todd
>
> On Mon, Jun 13, 2011 at 7:05 AM, Robert Evans <[EMAIL PROTECTED]> wrote:
>
>> Could someone unlock some of these branches for anonymous read only
>> checkout? At least with MR-279 I get a 403 forbidden error when I try to
>> check out.
>>
>> --Bobby
>>
>> On 6/12/11 6:38 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:
>>
>> OK, this seems to have succeeded without any big problems!
>>
>> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
>> to the new trees.
>>
>> Here are some instructions for the migration:
>>
>> === SVN users ==>>
>> Next time you "svn up" in your "common" working directory you'll end up
>> seeing the combined tree - ie a mapreduce/, hdfs/, and common/
>> subdirectory.
>> This is probably the easiest place from which to work, now. The URLs for
>> the
>> combined SVN trees are:
>>
>> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
>> branch-0.22:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
>> branch-0.21<http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22branch-0.21>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
>> yahoo-merge<http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21yahoo-merge>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>> (this one has the yahoo-merge branches from common, hdfs, and mapred)
>> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>> (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>>
>> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
>> Pre-project-split branches like branch-0.20 should have remained
>> untouched.
>>
>> You can proceed to delete your checkouts of the individual mapred and hdfs
>> trees, since they exist within the combined trees above. If for some
>> reason
>> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
>> its new location, you can use the following incantation:
>> svn sw $(svn info | grep URL | awk '{print $2}' | sed
>> 's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')
>>
>> === Git Users ==>> The git mirrors of the above 7 branches should now have a set of 4 commits
>> near the top that look like this:
>>
>> Merge: 928d485 cd66945 77f628f
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:28 2011 +0000
>>
>> HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
>> single tree (project unsplit)
>>
>> git-svn-id:
>> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-<https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68>
>> 0310-9956-ffa450edef68
>>
>> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:27 2011 +0000
>>
>> Relocate mapreduce into mapreduce/
>>
>> commit cd66945f62635f589ff93468e94c0039684a8b6d
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:26 2011 +0000
>>
>> Relocate hdfs into hdfs/
>>
>> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:25 2011 +0000
>>
>> Relocate common into common/
>>
>> The first of these 4 is a 3-parent "octopus" merge commit of the
>> pre-project-unsplit branches. In theory, git is smart enough to track
>> changes through this merge, so long as you pass the right flags (eg

Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Jeffrey Naisbitt 2011-06-13, 15:05

When I checkout the yahoo-merge branch, I see these svn externals warnings:
svn: warning: Error handling externals definition for
'yahoo-merge/hdfs/src/test/bin':
svn: warning: URL
'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
revision 1135120 doesn't exist
svn: warning: Error handling externals definition for
'yahoo-merge/mapreduce/src/test/bin':
svn: warning: URL
'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
revision 1135120 doesn't exist

Also, the ant eclipse targets seem to be broken now. It seems like various
parts of the eclipse target need to be commonized now (the
.eclipse-templates stuff and .classpath, .launches, etc.)

-Jeff

On 6/12/11 6:38 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:

> OK, this seems to have succeeded without any big problems!
>
> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
> to the new trees.
>
> Here are some instructions for the migration:
>
> === SVN users ==>
> Next time you "svn up" in your "common" working directory you'll end up
> seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirectory.
> This is probably the easiest place from which to work, now. The URLs for the
> combined SVN trees are:
>
> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> branch-0.22:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> branch-0.21:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> yahoo-merge:
> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
> (this one has the yahoo-merge branches from common, hdfs, and mapred)
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
> (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>
> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> Pre-project-split branches like branch-0.20 should have remained untouched.
>
> You can proceed to delete your checkouts of the individual mapred and hdfs
> trees, since they exist within the combined trees above. If for some reason
> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
> its new location, you can use the following incantation:
> svn sw $(svn info | grep URL | awk '{print $2}' | sed
> 's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')
>
> === Git Users ==> The git mirrors of the above 7 branches should now have a set of 4 commits
> near the top that look like this:
>
> Merge: 928d485 cd66945 77f628f
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:28 2011 +0000
>
> HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
> single tree (project unsplit)
>
> git-svn-id:
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310
> -9956-ffa450edef68
>
> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:27 2011 +0000
>
> Relocate mapreduce into mapreduce/
>
> commit cd66945f62635f589ff93468e94c0039684a8b6d
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:26 2011 +0000
>
> Relocate hdfs into hdfs/
>
> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> Author: Todd Lipcon <[EMAIL PROTECTED]>
> Date: Sun Jun 12 22:53:25 2011 +0000
>
> Relocate common into common/
>
> The first of these 4 is a 3-parent "octopus" merge commit of the
> pre-project-unsplit branches. In theory, git is smart enough to track
> changes through this merge, so long as you pass the right flags (eg
> --follow). For example:
>
> todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline --abbrev-commit
> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
> -10
> 77f628f Relocate mapreduce into mapreduce/
> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
> JobTrackerStatus.
> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
> aid diagnosis of related issues. Contributed by Jonathan Eagles

Re: HADOOP-7106 (project unsplit) this weekend

Robert Evans 2011-06-13, 15:10

Todd,

It seems to be working now, so whatever it was it is now gone.

--Bobby

On 6/13/11 9:57 AM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:

Hmm, as I got farther down my email this morning, I saw some complaints that
HBase's SVN was giving 403s as well, this morning. Perhaps there is some
ASF-wide issue going on, completely unrelated to the Hadoop changes over the
weekend?

-Todd

On Mon, Jun 13, 2011 at 7:52 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hey Robert,
>
> It seems to be working for me... what URL are you trying to check out?
> I moved aside my ~/.subversion dir, and then did:
> $ svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/
>
> Thanks
> -Todd
>
> On Mon, Jun 13, 2011 at 7:05 AM, Robert Evans <[EMAIL PROTECTED]> wrote:
>
>> Could someone unlock some of these branches for anonymous read only
>> checkout? At least with MR-279 I get a 403 forbidden error when I try to
>> check out.
>>
>> --Bobby
>>
>> On 6/12/11 6:38 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:
>>
>> OK, this seems to have succeeded without any big problems!
>>
>> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
>> to the new trees.
>>
>> Here are some instructions for the migration:
>>
>> === SVN users ==>>
>> Next time you "svn up" in your "common" working directory you'll end up
>> seeing the combined tree - ie a mapreduce/, hdfs/, and common/
>> subdirectory.
>> This is probably the easiest place from which to work, now. The URLs for
>> the
>> combined SVN trees are:
>>
>> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
>> branch-0.22:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
>> branch-0.21<http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22branch-0.21>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
>> yahoo-merge<http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21yahoo-merge>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>> (this one has the yahoo-merge branches from common, hdfs, and mapred)
>> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>> (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>>
>> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
>> Pre-project-split branches like branch-0.20 should have remained
>> untouched.
>>
>> You can proceed to delete your checkouts of the individual mapred and hdfs
>> trees, since they exist within the combined trees above. If for some
>> reason
>> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
>> its new location, you can use the following incantation:
>> svn sw $(svn info | grep URL | awk '{print $2}' | sed
>> 's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')
>>
>> === Git Users ==>> The git mirrors of the above 7 branches should now have a set of 4 commits
>> near the top that look like this:
>>
>> Merge: 928d485 cd66945 77f628f
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:28 2011 +0000
>>
>> HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
>> single tree (project unsplit)
>>
>> git-svn-id:
>> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-<https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68>
>> 0310-9956-ffa450edef68
>>
>> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:27 2011 +0000
>>
>> Relocate mapreduce into mapreduce/
>>
>> commit cd66945f62635f589ff93468e94c0039684a8b6d
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:26 2011 +0000
>>
>> Relocate hdfs into hdfs/
>>
>> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:25 2011 +0000
>>
>> Relocate common into common/
>>
>> The first of these 4 is a 3-parent "octopus" merge commit of the

Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-13, 15:14

On Mon, Jun 13, 2011 at 8:05 AM, Jeffrey Naisbitt <[EMAIL PROTECTED]>wrote:

> When I checkout the yahoo-merge branch, I see these svn externals warnings:
> svn: warning: Error handling externals definition for
> 'yahoo-merge/hdfs/src/test/bin':
> svn: warning: URL
> 'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
> revision 1135120 doesn't exist
> svn: warning: Error handling externals definition for
> 'yahoo-merge/mapreduce/src/test/bin':
> svn: warning: URL
> 'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
> revision 1135120 doesn't exist
>

Oops, sorry about that one. I will take care of that in about 30 minutes
(just headed out the door now to catch a train). If someone else with commit
access wants to, you just need to propset the externals to point to th new
common/trunk/common/src/test/bin instead of the old location.
> Also, the ant eclipse targets seem to be broken now. It seems like various
> parts of the eclipse target need to be commonized now (the
> .eclipse-templates stuff and .classpath, .launches, etc.)
>
>
Will look into this as well.

-Todd
>
> On 6/12/11 6:38 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:
>
> > OK, this seems to have succeeded without any big problems!
> >
> > I've re-enabled the git mirrors and the hudson builds. Feel free to
> commit
> > to the new trees.
> >
> > Here are some instructions for the migration:
> >
> > === SVN users ==> >
> > Next time you "svn up" in your "common" working directory you'll end up
> > seeing the combined tree - ie a mapreduce/, hdfs/, and common/
> subdirectory.
> > This is probably the easiest place from which to work, now. The URLs for
> the
> > combined SVN trees are:
> >
> > trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> > branch-0.22:
> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> > branch-0.21:
> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> > yahoo-merge:
> > http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
> > (this one has the yahoo-merge branches from common, hdfs, and mapred)
> > MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
> > (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
> >
> > The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> > Pre-project-split branches like branch-0.20 should have remained
> untouched.
> >
> > You can proceed to delete your checkouts of the individual mapred and
> hdfs
> > trees, since they exist within the combined trees above. If for some
> reason
> > you prefer to 'svn switch' an old MR or HDFS-specific checkout to point
> to
> > its new location, you can use the following incantation:
> > svn sw $(svn info | grep URL | awk '{print $2}' | sed
> > 's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')
> >
> > === Git Users ==> > The git mirrors of the above 7 branches should now have a set of 4
> commits
> > near the top that look like this:
> >
> > Merge: 928d485 cd66945 77f628f
> > Author: Todd Lipcon <[EMAIL PROTECTED]>
> > Date: Sun Jun 12 22:53:28 2011 +0000
> >
> > HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in
> a
> > single tree (project unsplit)
> >
> > git-svn-id:
> >
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310
> > -9956-ffa450edef68
> >
> > commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> > Author: Todd Lipcon <[EMAIL PROTECTED]>
> > Date: Sun Jun 12 22:53:27 2011 +0000
> >
> > Relocate mapreduce into mapreduce/
> >
> > commit cd66945f62635f589ff93468e94c0039684a8b6d
> > Author: Todd Lipcon <[EMAIL PROTECTED]>
> > Date: Sun Jun 12 22:53:26 2011 +0000
> >
> > Relocate hdfs into hdfs/
> >
> > commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> > Author: Todd Lipcon <[EMAIL PROTECTED]>
> > Date: Sun Jun 12 22:53:25 2011 +0000
> >
> > Relocate common into common/
> >
> > The first of these 4 is a 3-parent "octopus" merge commit of the
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Nigel Daley 2011-06-13, 15:35

Yes, many thanks for carrying this over the finish line!

Cheers,
Nige

Sent from my iPhone4

On Jun 13, 2011, at 12:39 AM, Eli Collins <[EMAIL PROTECTED]> wrote:

> Nice work Todd. Thank you for making this happen!
>
>
> On Sun, Jun 12, 2011 at 4:38 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>> OK, this seems to have succeeded without any big problems!
>>
>> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
>> to the new trees.
>>
>> Here are some instructions for the migration:
>>
>> === SVN users ==>>
>> Next time you "svn up" in your "common" working directory you'll end up
>> seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirectory.
>> This is probably the easiest place from which to work, now. The URLs for the
>> combined SVN trees are:
>>
>> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
>> branch-0.22:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
>> branch-0.21:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
>> yahoo-merge:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>> (this one has the yahoo-merge branches from common, hdfs, and mapred)
>> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>> (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>>
>> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
>> Pre-project-split branches like branch-0.20 should have remained untouched.
>>
>> You can proceed to delete your checkouts of the individual mapred and hdfs
>> trees, since they exist within the combined trees above. If for some reason
>> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
>> its new location, you can use the following incantation:
>> svn sw $(svn info | grep URL | awk '{print $2}' | sed
>> 's,$hdfs\|mapreduce\|common$/$.*$,common/\2/\1,')
>>
>> === Git Users ==>> The git mirrors of the above 7 branches should now have a set of 4 commits
>> near the top that look like this:
>>
>> Merge: 928d485 cd66945 77f628f
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:28 2011 +0000
>>
>> HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
>> single tree (project unsplit)
>>
>> git-svn-id:
>> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68
>>
>> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:27 2011 +0000
>>
>> Relocate mapreduce into mapreduce/
>>
>> commit cd66945f62635f589ff93468e94c0039684a8b6d
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:26 2011 +0000
>>
>> Relocate hdfs into hdfs/
>>
>> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
>> Author: Todd Lipcon <[EMAIL PROTECTED]>
>> Date: Sun Jun 12 22:53:25 2011 +0000
>>
>> Relocate common into common/
>>
>> The first of these 4 is a 3-parent "octopus" merge commit of the
>> pre-project-unsplit branches. In theory, git is smart enough to track
>> changes through this merge, so long as you pass the right flags (eg
>> --follow). For example:
>>
>> todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline --abbrev-commit
>> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
>> -10
>> 77f628f Relocate mapreduce into mapreduce/
>> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
>> JobTrackerStatus.
>> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
>> aid diagnosis of related issues. Contributed by Jonathan Eagles
>> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
>> Contributed by Ari Rabkin.
>>
>> If you want to be able to have git follow renames all the way through the
>> project split back to the beginning of time, put the following in
>> hadoop-common/.git/info/grafts:
>> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-13, 16:03

On Mon, Jun 13, 2011 at 8:14 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

>
> Oops, sorry about that one. I will take care of that in about 30 minutes
> (just headed out the door now to catch a train). If someone else with commit
> access wants to, you just need to propset the externals to point to th new
> common/trunk/common/src/test/bin instead of the old location.
>
>
Fixed the svn:externals
>
>> Also, the ant eclipse targets seem to be broken now. It seems like
>> various
>> parts of the eclipse target need to be commonized now (the
>> .eclipse-templates stuff and .classpath, .launches, etc.)
>>
>>
> Will look into this as well.
>
> Can you explain further what's broken? Are you trying to make a project
that's rooted in the directory that contains common/, mapreduce/, and hdfs/?
I can imagine that wouldn't work, but I'm not sure why it wouldn't work to
continue having three separate projects. Are you using some kind of SVN
integration with Eclipse?

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Jeffrey Naisbitt 2011-06-13, 16:24

On 6/13/11 11:03 AM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:

> On Mon, Jun 13, 2011 at 8:14 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Fixed the svn:externals

Thanks! :)

>>
>>> Also, the ant eclipse targets seem to be broken now. It seems like
>>> various
>>> parts of the eclipse target need to be commonized now (the
>>> .eclipse-templates stuff and .classpath, .launches, etc.)
>>>
>>>
>> Will look into this as well.
>>
>> Can you explain further what's broken? Are you trying to make a project
> that's rooted in the directory that contains common/, mapreduce/, and hdfs/?
> I can imagine that wouldn't work, but I'm not sure why it wouldn't work to
> continue having three separate projects. Are you using some kind of SVN
> integration with Eclipse?

>>> Also, the ant eclipse targets seem to be broken now. It seems like
>>> various
>>> parts of the eclipse target need to be commonized now (the
>>> .eclipse-templates stuff and .classpath, .launches, etc.)
>>>
>>>
>> Will look into this as well.
>>
>> Can you explain further what's broken? Are you trying to make a project
> that's rooted in the directory that contains common/, mapreduce/, and hdfs/?
> I can imagine that wouldn't work, but I'm not sure why it wouldn't work to
> continue having three separate projects. Are you using some kind of SVN
> integration with Eclipse?

I am using subversive for svn integration. I was actually trying to use a
single project for the whole thing
(https://svn.apache.org/repos/asf/hadoop/common/trunk)
As you say though, I would think it should still work with three separate
projects, so I'll just go back to that for now. It would be nice to be able
to build the whole thing as a single project though (since it's a single
repository in svn now), but that would probably take some extra work - and
would probably make sense in a separate Jira.

Thanks.
-Jeff

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-13, 17:01

On Mon, Jun 13, 2011 at 9:24 AM, Jeffrey Naisbitt <[EMAIL PROTECTED]>wrote:

> As you say though, I would think it should still work with three separate
> projects, so I'll just go back to that for now. It would be nice to be
> able
> to build the whole thing as a single project though (since it's a single
> repository in svn now), but that would probably take some extra work - and
> would probably make sense in a separate Jira.
>

Yep - I think the idea is this will happen once we mavenize everything.
Maven apparently supports the idea of "modules" - basically a recursive
structure in which a top level pom file can build sub-poms, but the subpoms
could also continue to be built independently if necessary. I think the
top-level pom would live in the new root.

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Alejandro Abdelnur 2011-06-13, 17:04

On Mon, Jun 13, 2011 at 10:01 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Mon, Jun 13, 2011 at 9:24 AM, Jeffrey Naisbitt <[EMAIL PROTECTED]
> >wrote:
>
> > As you say though, I would think it should still work with three separate
> > projects, so I'll just go back to that for now. It would be nice to be
> > able
> > to build the whole thing as a single project though (since it's a single
> > repository in svn now), but that would probably take some extra work -
> and
> > would probably make sense in a separate Jira.
> >
>
> Yep - I think the idea is this will happen once we mavenize everything.
> Maven apparently supports the idea of "modules" - basically a recursive
> structure in which a top level pom file can build sub-poms, but the subpoms
> could also continue to be built independently if necessary. I think the
> top-level pom would live in the new root.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
That sounds just right!

Thxs.

Alejandro

Re: HADOOP-7106 (project unsplit) this weekend

Jeffrey Naisbitt 2011-06-13, 17:08

On 6/13/11 12:01 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:

> On Mon, Jun 13, 2011 at 9:24 AM, Jeffrey Naisbitt
> <[EMAIL PROTECTED]>wrote:
>
>> As you say though, I would think it should still work with three separate
>> projects, so I'll just go back to that for now. It would be nice to be
>> able
>> to build the whole thing as a single project though (since it's a single
>> repository in svn now), but that would probably take some extra work - and
>> would probably make sense in a separate Jira.
>>
>
> Yep - I think the idea is this will happen once we mavenize everything.
> Maven apparently supports the idea of "modules" - basically a recursive
> structure in which a top level pom file can build sub-poms, but the subpoms
> could also continue to be built independently if necessary. I think the
> top-level pom would live in the new root.
>
> -Todd

I look forward to the mavenization! :)
I verified the eclipse builds are working fine for me in separate projects
and that the svn:externals work fine on yahoo-merge now.
Thanks for the quick responses and for all the other work on the "unsplit"!
-Jeff

Re: HADOOP-7106 (project unsplit) this weekend

Tsz Wo \ 2011-06-13, 18:42

Todd,

Great work!
A few minor problems:
(1) I had committed MAPREDUCE-2588, however, an commit email was sent to both
common-commits@ and mapreduce-commits@.
(2) hadoop/site becomes an empty directory.
(3) There are svn properties in hadoop/common/trunk/

(2) and (3) are simple. I will remove hadoop/site and the svn properties for
hadoop/common/trunk/.

Does anyone know how to fix (1)?
A separated question:
Strictly speaking, this is not "project unsplit" since we are going to submit
patches for individual sub-projects as before, i.e. we keep generating and
committing patches to common/trunk/[common/hdfs/mapreduce] but not
common/trunk; hudson will pick up patches from individual sub-projects, etc.
Am I correct? Just want to make sure that everyone is on the same page. :)
Regards,
Tsz-Wo

Re: HADOOP-7106 (project unsplit) this weekend

Matthew Foley 2011-06-13, 19:51

Todd, it's so great to have this done. Thanks for the huge amount of hard work!
--Matt

On Jun 13, 2011, at 10:08 AM, Jeffrey Naisbitt wrote:

On 6/13/11 12:01 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:

> On Mon, Jun 13, 2011 at 9:24 AM, Jeffrey Naisbitt
> <[EMAIL PROTECTED]>wrote:
>
>> As you say though, I would think it should still work with three separate
>> projects, so I'll just go back to that for now. It would be nice to be
>> able
>> to build the whole thing as a single project though (since it's a single
>> repository in svn now), but that would probably take some extra work - and
>> would probably make sense in a separate Jira.
>>
>
> Yep - I think the idea is this will happen once we mavenize everything.
> Maven apparently supports the idea of "modules" - basically a recursive
> structure in which a top level pom file can build sub-poms, but the subpoms
> could also continue to be built independently if necessary. I think the
> top-level pom would live in the new root.
>
> -Todd

I look forward to the mavenization! :)
I verified the eclipse builds are working fine for me in separate projects
and that the svn:externals work fine on yahoo-merge now.
Thanks for the quick responses and for all the other work on the "unsplit"!
-Jeff

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-13, 20:35

On Mon, Jun 13, 2011 at 11:42 AM, Tsz Wo (Nicholas), Sze <
[EMAIL PROTECTED]> wrote:

> Todd,
>
> Great work!
>
>
> A few minor problems:
> (1) I had committed MAPREDUCE-2588, however, an commit email was sent to
> both
> common-commits@ and mapreduce-commits@.
> (2) hadoop/site becomes an empty directory.
> (3) There are svn properties in hadoop/common/trunk/
>
> (2) and (3) are simple. I will remove hadoop/site and the svn properties
> for
> hadoop/common/trunk/.
>
> Does anyone know how to fix (1)?
>

Ah, we need to update the mailer config. This is the remaining task I
mentioned on the HADOOP-7106 JIRA. We had updated it to include both the old
and new spots for the sake of the transition. I'll ping Ian about this - I
think he's the only one with access.
>
>
> A separated question:
> Strictly speaking, this is not "project unsplit" since we are going to
> submit
> patches for individual sub-projects as before, i.e. we keep generating and
> committing patches to common/trunk/[common/hdfs/mapreduce] but not
> common/trunk; hudson will pick up patches from individual sub-projects,
> etc.
> Am I correct? Just want to make sure that everyone is on the same page.
> :)
>
>
Correct, that's where we at for today. I opened HADOOP-7384 which will allow
patches generated against the "full repo" to apply to an individual project,
to make life easier for git users, but right now we still need separate
JIRAs and patches. See my other thread from this morning about some ideas
how we might be able to do cross-project patches in a reasonable way.

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-13, 20:52

On Sun, Jun 12, 2011 at 4:38 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> If you want to be able to have git follow renames all the way through the
> project split back to the beginning of time, put the following in
> hadoop-common/.git/info/grafts:
> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 546d96754ffee3142bcbbf4563c624c053d0ed0d
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>
>
ATM has pointed out that the above contents line-wrapped in some people's
email inboxes. I've put the correct information on the GitAndHadoop wiki
page here:
http://wiki.apache.org/hadoop/GitAndHadoop

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-14, 01:49

On Mon, Jun 13, 2011 at 11:42 AM, Tsz Wo (Nicholas), Sze <
[EMAIL PROTECTED]> wrote:

> A few minor problems:
> (1) I had committed MAPREDUCE-2588, however, an commit email was sent to
> both
> common-commits@ and mapreduce-commits@.
>

I put up a patch to fix this on HADOOP-7106. Unfortunately I think Doug and
Ian are the only ones who can commit it, and they're both traveling at the
moment. So we may have to endure dup emails for a day or two. Sorry for the
mailbox noise

-Todd

--
Todd Lipcon
Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Ian Holsman 2011-06-14, 23:32

On Jun 14, 2011, at 11:49 AM, Todd Lipcon wrote:

> On Mon, Jun 13, 2011 at 11:42 AM, Tsz Wo (Nicholas), Sze <
> [EMAIL PROTECTED]> wrote:
>
>> A few minor problems:
>> (1) I had committed MAPREDUCE-2588, however, an commit email was sent to
>> both
>> common-commits@ and mapreduce-commits@.
>>
>
> I put up a patch to fix this on HADOOP-7106. Unfortunately I think Doug and
> Ian are the only ones who can commit it, and they're both traveling at the
> moment. So we may have to endure dup emails for a day or two. Sorry for the
> mailbox noise
>

should be fixed now.
(back home, nothing quite like sleeping in your own bed)

> -Todd
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

Re: HADOOP-7106 (project unsplit) this weekend

Ranjit Mathew 2011-06-16, 09:27

On 06/13/2011 09:33 PM, Todd Lipcon wrote:
>> Oops, sorry about that one. I will take care of that in about 30 minutes
>> (just headed out the door now to catch a train). If someone else with commit
>> access wants to, you just need to propset the externals to point to th new
>> common/trunk/common/src/test/bin instead of the old location.
>>
>>
> Fixed the svn:externals

Does it make sense for "test-patch.sh" and "smart-apply-patch.sh" to
remain "external" now that they're within the same project?

Currently on every "svn up" for trunk I get:
-------------------------- 8< --------------------------
$ svn up

Fetching external item into 'hdfs/src/test/bin'
External at revision 1136333.
Fetching external item into 'mapreduce/src/test/bin'
External at revision 1136333.

At revision 1136333.
-------------------------- 8< --------------------------

after a little pause.

Ranjit

PS: I'm using "svn, version 1.6.16 (r1073529)" on Fedora 14/x86-32.

Re: HADOOP-7106 (project unsplit) this weekend

Todd Lipcon 2011-06-16, 16:01

On Thu, Jun 16, 2011 at 2:27 AM, Ranjit Mathew <[EMAIL PROTECTED]> wrote:

> On 06/13/2011 09:33 PM, Todd Lipcon wrote:
>
>> Oops, sorry about that one. I will take care of that in about 30 minutes
>>> (just headed out the door now to catch a train). If someone else with
>>> commit
>>> access wants to, you just need to propset the externals to point to th
>>> new
>>> common/trunk/common/src/test/**bin instead of the old location.
>>>
>>>
>>> Fixed the svn:externals
>>
>
> Does it make sense for "test-patch.sh" and "smart-apply-patch.sh" to
> remain "external" now that they're within the same project?
>

Probably not long-term... but right now the various subproject hudson builds
only check out the hdfs/ or mapreduce/ subtree, so rooting the test scripts
above that level would require a bit of reconfiguration.
>
> Currently on every "svn up" for trunk I get:
> -------------------------- 8< --------------------------
> $ svn up
>
> Fetching external item into 'hdfs/src/test/bin'
> External at revision 1136333.
>
>
> Fetching external item into 'mapreduce/src/test/bin'
> External at revision 1136333.
>
> At revision 1136333.
> -------------------------- 8< --------------------------
>
> after a little pause.
>
> Ranjit
>
> PS: I'm using "svn, version 1.6.16 (r1073529)" on Fedora 14/x86-32.
>

--
Todd Lipcon
Software Engineer, Cloudera