All,

As discussed in Jan/Feb, I'd like to coordinate a date for committing the re-organization of our svn layout: https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday April 21 at 11am PDT.

- I will send out reminders leading up to that date.
- I will announce on IRC when I'm about to start the changes.
- I will run the script to make the changes.
- Ian, can you update the asf-authorization-template file and the asf-mailer.conf files at the same time?
- Owen/Todd/Jukka, can you make sure that actions needed by git users are taken care of at the same time? (what are these)

More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit

Cheers,
Nige

I'm still planning to make this SVN change on Thursday this week.  

Ian, Owen, Todd, note the questions I ask you below.  Can you help with these on Thursday?

Thanks,
Nige

On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:

> All,
>
> As discussed in Jan/Feb, I'd like to coordinate a date for committing the re-organization of our svn layout: https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday April 21 at 11am PDT.
>
> - I will send out reminders leading up to that date.
> - I will announce on IRC when I'm about to start the changes.
> - I will run the script to make the changes.
> - Ian, can you update the asf-authorization-template file and the asf-mailer.conf files at the same time?
> - Owen/Todd/Jukka, can you make sure that actions needed by git users are taken care of at the same time? (what are these?)
>
> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
>
> Cheers,
> Nige

On Tue, Apr 19, 2011 at 10:02 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:

> I'm still planning to make this SVN change on Thursday this week.
>
> Ian, Owen, Todd, note the questions I ask you below.  Can you help with
> these on Thursday?
>

Unfortunately I'm out of the office most of the day on Thursday with a
customer. I'll be available Thursday evening, though, to help with any
cleanup/etc.

I'm currently looking into how the git mirrors are setup in Apache-land.

My guess is that there will be some disturbance to developers on Thurs
afternoon / Friday as this gets sorted out, even if we try to plan as much
as possible. Would it be better to do this on Friday so that we have the
weekend to fix up broken pieces before people get to work on Monday?

-Todd
> On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:
>
> All,
>
> As discussed in Jan/Feb, I'd like to coordinate a date for committing the
> re-organization of our svn layout:
> https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday
> April 21 at 11am PDT.
>
> - I will send out reminders leading up to that date.
> - I will announce on IRC when I'm about to start the changes.
> - I will run the script to make the changes.
> - Ian, can you update the asf-authorization-template file and the
> asf-mailer.conf files at the same time?
> - Owen/Todd/Jukka, can you make sure that actions needed by git users are
> taken care of at the same time? (what are these?)
>
> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
>
> Cheers,
> Nige
>
>
>
--
Todd Lipcon
Software Engineer, Cloudera

I could start this at 2pm on Friday if that suited folks better.

Nige

On Apr 19, 2011, at 10:20 PM, Todd Lipcon wrote:

> On Tue, Apr 19, 2011 at 10:02 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:
>
>> I'm still planning to make this SVN change on Thursday this week.
>>
>> Ian, Owen, Todd, note the questions I ask you below.  Can you help with
>> these on Thursday?
>>
>
> Unfortunately I'm out of the office most of the day on Thursday with a
> customer. I'll be available Thursday evening, though, to help with any
> cleanup/etc.
>
> I'm currently looking into how the git mirrors are setup in Apache-land.
>
> My guess is that there will be some disturbance to developers on Thurs
> afternoon / Friday as this gets sorted out, even if we try to plan as much
> as possible. Would it be better to do this on Friday so that we have the
> weekend to fix up broken pieces before people get to work on Monday?
>
> -Todd
>
>
>> On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:
>>
>> All,
>>
>> As discussed in Jan/Feb, I'd like to coordinate a date for committing the
>> re-organization of our svn layout:
>> https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday
>> April 21 at 11am PDT.
>>
>> - I will send out reminders leading up to that date.
>> - I will announce on IRC when I'm about to start the changes.
>> - I will run the script to make the changes.
>> - Ian, can you update the asf-authorization-template file and the
>> asf-mailer.conf files at the same time?
>> - Owen/Todd/Jukka, can you make sure that actions needed by git users are
>> taken care of at the same time? (what are these?)
>>
>> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
>>
>> Cheers,
>> Nige
>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

>
> I'm currently looking into how the git mirrors are setup in Apache-land.
>

Git-wise, I think we have two options:

Option 1)
- Create a new git mirror for the new hadoop/ tree. This will have no
history.
- On the Apache side, fetch the split-project git mirrors into the combined
git mirror as branches - eg hadoop-hdfs.git:trunk becomes a branch named
something like pre-HADOOP-7106/hdfs/trunk. Thus, when any user fetches,
he'll get all the git objects from "prehistory" as well without having to
add separate remotes.
- Add a script or README file explaining how to set up git grafts on the
combined hadoop.git so that the new combination branch "foo" looks like a
merge of pre-HADOOP-7106/{hdfs,common,mapred}/foo. Since git grafts are
local constructs, each git user would have to run this script once after
checking out the git tree, after which the history would be "healed"

Pros:
 - all existing sha1s stay the same.
 - Any local branches people might have for works in progress should
continue to refer to proper SHA1s and should rebase relatively easily onto
the combined trunk
 - Should be reasonably simple to implement

Cons:
 - users have to run a script upon checkout in order to graft back together
history

Option 2)
- Use git-filter-branch on the split repos to rewrite them as if they always
took place in their new subdirectories.
- Fetch these repos into the merged repo
- Set up grafts in the merged repo
- Run git-filter-branch --all in the merged repo, which will make the grafts
permanent
- May have to run git-filter-branch to rewrite some of the git-svn-info:
commit messages to trick git-svn.

This option basically rewrites history so that it looks like the original
project split did what we're planning to do now.

Pros:
 - we have a single cohesive git repo with no need to have users set up
grafts

Cons:
 - all of our SHA1s between the original split and now would change (making
it harder to rebase local branches for example)
 - way more opportunity for error, I think.

I'm leaning towards option 1 above, and happy to write the script which
installs the grafts into the user's local repo.

-Todd
>
>> On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:
>>
>> All,
>>
>> As discussed in Jan/Feb, I'd like to coordinate a date for committing the
>> re-organization of our svn layout:
>> https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday
>> April 21 at 11am PDT.
>>
>> - I will send out reminders leading up to that date.
>> - I will announce on IRC when I'm about to start the changes.
>> - I will run the script to make the changes.
>> - Ian, can you update the asf-authorization-template file and the
>> asf-mailer.conf files at the same time?
>> - Owen/Todd/Jukka, can you make sure that actions needed by git users are
>> taken care of at the same time? (what are these?)
>>
>> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
>>
>> Cheers,
>> Nige
>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--
Todd Lipcon
Software Engineer, Cloudera

I'm traveling at the moment (when am I not), and probably won't have access until monday.
I'll look at the files, and see if I can do some stuff today without impacting anything.
On Apr 20, 2011, at 6:58 AM, Todd Lipcon wrote:

> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
> I'm currently looking into how the git mirrors are setup in Apache-land.
>
> Git-wise, I think we have two options:
>
> Option 1)
> - Create a new git mirror for the new hadoop/ tree. This will have no history.
> - On the Apache side, fetch the split-project git mirrors into the combined git mirror as branches - eg hadoop-hdfs.git:trunk becomes a branch named something like pre-HADOOP-7106/hdfs/trunk. Thus, when any user fetches, he'll get all the git objects from "prehistory" as well without having to add separate remotes.
> - Add a script or README file explaining how to set up git grafts on the combined hadoop.git so that the new combination branch "foo" looks like a merge of pre-HADOOP-7106/{hdfs,common,mapred}/foo. Since git grafts are local constructs, each git user would have to run this script once after checking out the git tree, after which the history would be "healed"
>
> Pros:
>  - all existing sha1s stay the same.
>  - Any local branches people might have for works in progress should continue to refer to proper SHA1s and should rebase relatively easily onto the combined trunk
>  - Should be reasonably simple to implement
>
> Cons:
>  - users have to run a script upon checkout in order to graft back together history
>
> Option 2)
> - Use git-filter-branch on the split repos to rewrite them as if they always took place in their new subdirectories.
> - Fetch these repos into the merged repo
> - Set up grafts in the merged repo
> - Run git-filter-branch --all in the merged repo, which will make the grafts permanent
> - May have to run git-filter-branch to rewrite some of the git-svn-info: commit messages to trick git-svn.
>
> This option basically rewrites history so that it looks like the original project split did what we're planning to do now.
>
> Pros:
>  - we have a single cohesive git repo with no need to have users set up grafts
>
> Cons:
>  - all of our SHA1s between the original split and now would change (making it harder to rebase local branches for example)
>  - way more opportunity for error, I think.
>
> I'm leaning towards option 1 above, and happy to write the script which installs the grafts into the user's local repo.
>
> -Todd
>
>  
> On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:
>
>> All,
>>
>> As discussed in Jan/Feb, I'd like to coordinate a date for committing the re-organization of our svn layout: https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday April 21 at 11am PDT.
>>
>> - I will send out reminders leading up to that date.
>> - I will announce on IRC when I'm about to start the changes.
>> - I will run the script to make the changes.
>> - Ian, can you update the asf-authorization-template file and the asf-mailer.conf files at the same time?
>> - Owen/Todd/Jukka, can you make sure that actions needed by git users are taken care of at the same time? (what are these?)
>>
>> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
>>
>> Cheers,
>> Nige
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

--
Ian Holsman
[EMAIL PROTECTED]
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

To know recursion, you must first know recursion.

On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:

> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>>
>> I'm currently looking into how the git mirrors are setup in Apache-land.

Uh, why isn't infra-dev on this thread?

For those on infra-dev, the context is that Nigel is trying to merge together the source trees of the Hadoop sub-projects that were split apart 2 years ago. So he is taking:

prefix = http://svn.apache.org/repos/asf/hadoop/

$prefix/common/trunk -> $prefix/trunk/common
$prefix/hdfs/trunk -> $prefix/trunk/hdfs
$prefix/mapreduce/trunk -> $prefix/trunk/mapreduce

and play similar games with the rest of the branches and tags. For more details look at HADOOP-7106.

From the project split, subversion was able to track the history across the subversion moves between projects, but not git.

Four questions:
 1. Is there anything we can do to minimize the history loss in git?
 2. Are we going to be able to preserve our sha's or are they going to change again?
 3. What changes do we need to make to the subversion notification file?
 4. Are there any other changes that need to be coordinated?

After considering it this morning, I believe that the least disruptive move is to leave common at the same url and merge hdfs and mapreduce back in:

$prefix/common/trunk/* -> $prefix/common/trunk/common/*
$prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
$prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce

This will preserve the hashes and history for common (and the 20 branches). We'll still need to play git voodoo to get git history for hdfs and mapreduce, but it is far better than starting a brand new git clone.

-- Owen

> From the project split, subversion was able to track the history across the subversion moves between projects, but not git.
>
> Four questions:
>  1. Is there anything we can do to minimize the history loss in git?

Glancing at the Git history on GitHub for Hadoop HDFS my guess is that
the mirror script is just pulling from $prefix/hdfs. I'm not entirely
certain how promiscuous it'll look for movements from elsewhere in the
svn repo for history outside the root URL.

If you're really wanting to make sure to keep the history in Git
intact my suggestion would be to setup a temporary svn server locally
and test our mirroring scripts against the commands you intend to run.

Paul Davis

On Wed, Apr 20, 2011 at 08:00, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> After considering it this morning, I believe that the least disruptive move is to leave common at the same url and merge hdfs and mapreduce back in:
>
> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce

This seems like adding an insult to injury by creating an artificial
SVN layout and moving what deemed to be an unrelated components into
common's namespace.

The original proposal seems much more straight forward.

Cos

> This will preserve the hashes and history for common (and the 20 branches). We'll still need to play git voodoo to get git history for hdfs and mapreduce, but it is far better than starting a brand new git clone.
>
> -- Owen

>

the other issue is when you create a branch, as you would need to update the SVN/mail configs to add each branch independently.. unless you can do wildcard matching in the config files.. infra??? can you do this?

On Apr 20, 2011, at 8:26 PM, Konstantin Boudnik wrote:

> On Wed, Apr 20, 2011 at 08:00, Owen O'Malley <[EMAIL PROTECTED]> wrote:
>> After considering it this morning, I believe that the least disruptive move is to leave common at the same url and merge hdfs and mapreduce back in:
>>
>> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
>> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
>> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
>
> This seems like adding an insult to injury by creating an artificial
> SVN layout and moving what deemed to be an unrelated components into
> common's namespace.
>
> The original proposal seems much more straight forward.
>
> Cos
>
>> This will preserve the hashes and history for common (and the 20 branches). We'll still need to play git voodoo to get git history for hdfs and mapreduce, but it is far better than starting a brand new git clone.
>>
>> -- Owen
>
>>

--
Ian Holsman
[EMAIL PROTECTED]
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

“If we knew what it was we were doing, it would not be called research, would it?”
 – Albert Einstein

Owen, I'll admit I'm not familiar with all the git details/issues in your proposal, but I think the layout change you propose is fine and seems to solve the git issues with very minimal impact on the layout.

Let's shoot for doing this next Friday, April 29 at 2pm PDT.  I'll update the patch and send out a reminder about this later next week.

Thanks,
Nige

On Apr 20, 2011, at 8:00 AM, Owen O'Malley wrote:

>
> On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:
>
>> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>
>>>
>>> I'm currently looking into how the git mirrors are setup in Apache-land.
>
> Uh, why isn't infra-dev on this thread?
>
> For those on infra-dev, the context is that Nigel is trying to merge together the source trees of the Hadoop sub-projects that were split apart 2 years ago. So he is taking:
>
> prefix = http://svn.apache.org/repos/asf/hadoop/
>
> $prefix/common/trunk -> $prefix/trunk/common
> $prefix/hdfs/trunk -> $prefix/trunk/hdfs
> $prefix/mapreduce/trunk -> $prefix/trunk/mapreduce
>
> and play similar games with the rest of the branches and tags. For more details look at HADOOP-7106.
>
> From the project split, subversion was able to track the history across the subversion moves between projects, but not git.
>
> Four questions:
> 1. Is there anything we can do to minimize the history loss in git?
> 2. Are we going to be able to preserve our sha's or are they going to change again?
> 3. What changes do we need to make to the subversion notification file?
> 4. Are there any other changes that need to be coordinated?
>
> After considering it this morning, I believe that the least disruptive move is to leave common at the same url and merge hdfs and mapreduce back in:
>
> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
>
> This will preserve the hashes and history for common (and the 20 branches). We'll still need to play git voodoo to get git history for hdfs and mapreduce, but it is far better than starting a brand new git clone.
>
> -- Owen
>
>

As announced last week, I'm planning to do this at 2pm PDT tomorrow (Friday) April 29.  

Suresh, when do you plan to commit HFS-1052?  That should be done first.

Owen or Todd, did you want to follow Paul's advice:
> If you're really wanting to make sure to keep the history in Git
> intact my suggestion would be to setup a temporary svn server locally
> and test our mirroring scripts against the commands you intend to run.
If so, how much more time do you need?

Cheers,
Nige

On Apr 20, 2011, at 9:42 PM, Nigel Daley wrote:

> Owen, I'll admit I'm not familiar with all the git details/issues in your proposal, but I think the layout change you propose is fine and seems to solve the git issues with very minimal impact on the layout.
>
> Let's shoot for doing this next Friday, April 29 at 2pm PDT.  I'll update the patch and send out a reminder about this later next week.
>
> Thanks,
> Nige
>
> On Apr 20, 2011, at 8:00 AM, Owen O'Malley wrote:
>
>>
>> On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:
>>
>>> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>>
>>>>
>>>> I'm currently looking into how the git mirrors are setup in Apache-land.
>>
>> Uh, why isn't infra-dev on this thread?
>>
>> For those on infra-dev, the context is that Nigel is trying to merge together the source trees of the Hadoop sub-projects that were split apart 2 years ago. So he is taking:
>>
>> prefix = http://svn.apache.org/repos/asf/hadoop/
>>
>> $prefix/common/trunk -> $prefix/trunk/common
>> $prefix/hdfs/trunk -> $prefix/trunk/hdfs
>> $prefix/mapreduce/trunk -> $prefix/trunk/mapreduce
>>
>> and play similar games with the rest of the branches and tags. For more details look at HADOOP-7106.
>>
>> From the project split, subversion was able to track the history across the subversion moves between projects, but not git.
>>
>> Four questions:
>> 1. Is there anything we can do to minimize the history loss in git?
>> 2. Are we going to be able to preserve our sha's or are they going to change again?
>> 3. What changes do we need to make to the subversion notification file?
>> 4. Are there any other changes that need to be coordinated?
>>
>> After considering it this morning, I believe that the least disruptive move is to leave common at the same url and merge hdfs and mapreduce back in:
>>
>> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
>> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
>> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
>>
>> This will preserve the hashes and history for common (and the 20 branches). We'll still need to play git voodoo to get git history for hdfs and mapreduce, but it is far better than starting a brand new git clone.
>>
>> -- Owen
>>
>>
>

On Thu, Apr 28, 2011 at 10:06 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:

> As announced last week, I'm planning to do this at 2pm PDT tomorrow
> (Friday) April 29.
>
> Suresh, when do you plan to commit HFS-1052?  That should be done first.
>
> Owen or Todd, did you want to follow Paul's advice:
> > If you're really wanting to make sure to keep the history in Git
> > intact my suggestion would be to setup a temporary svn server locally
> > and test our mirroring scripts against the commands you intend to run.
> If so, how much more time do you need?
>

Wasn't sure how to go about doing that. I guess we need to talk to infra
about it? Do you know how we might clone the SVN repos themselves to test
with?

-Todd

On Apr 20, 2011, at 9:42 PM, Nigel Daley wrote:
>
> > Owen, I'll admit I'm not familiar with all the git details/issues in your
> proposal, but I think the layout change you propose is fine and seems to
> solve the git issues with very minimal impact on the layout.
> >
> > Let's shoot for doing this next Friday, April 29 at 2pm PDT.  I'll update
> the patch and send out a reminder about this later next week.
> >
> > Thanks,
> > Nige
> >
> > On Apr 20, 2011, at 8:00 AM, Owen O'Malley wrote:
> >
> >>
> >> On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:
> >>
> >>> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>>
> >>>> I'm currently looking into how the git mirrors are setup in
> Apache-land.
> >>
> >> Uh, why isn't infra-dev on this thread?
> >>
> >> For those on infra-dev, the context is that Nigel is trying to merge
> together the source trees of the Hadoop sub-projects that were split apart 2
> years ago. So he is taking:
> >>
> >> prefix = http://svn.apache.org/repos/asf/hadoop/
> >>
> >> $prefix/common/trunk -> $prefix/trunk/common
> >> $prefix/hdfs/trunk -> $prefix/trunk/hdfs
> >> $prefix/mapreduce/trunk -> $prefix/trunk/mapreduce
> >>
> >> and play similar games with the rest of the branches and tags. For more
> details look at HADOOP-7106.
> >>
> >> From the project split, subversion was able to track the history across
> the subversion moves between projects, but not git.
> >>
> >> Four questions:
> >> 1. Is there anything we can do to minimize the history loss in git?
> >> 2. Are we going to be able to preserve our sha's or are they going to
> change again?
> >> 3. What changes do we need to make to the subversion notification file?
> >> 4. Are there any other changes that need to be coordinated?
> >>
> >> After considering it this morning, I believe that the least disruptive
> move is to leave common at the same url and merge hdfs and mapreduce back
> in:
> >>
> >> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
> >> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
> >> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
> >>
> >> This will preserve the hashes and history for common (and the 20
> branches). We'll still need to play git voodoo to get git history for hdfs
> and mapreduce, but it is far better than starting a brand new git clone.
> >>
> >> -- Owen
> >>
> >>
> >
>
>
--
Todd Lipcon
Software Engineer, Cloudera

On Apr 28, 2011, at 11:24 PM, Todd Lipcon wrote:

> Wasn't sure how to go about doing that. I guess we need to talk to infra about it? Do you know how we might clone the SVN repos themselves to test with?

It looks like there are svn dumps at http://svn-master.apache.org/dump/ from 2 april 2011. You should be able to use those to setup a local subversion.

-- Owen

I can't do this at 2pm now.  Todd, I suspect you want more time to try out the svn/git test anyways.

Let's shoot for next Wednesday at 2pm.  Ian should be back by then too.  Any objections?

Cheers,
Nige

On Apr 29, 2011, at 11:36 AM, Owen O'Malley wrote:

>
> On Apr 28, 2011, at 11:24 PM, Todd Lipcon wrote:
>
>> Wasn't sure how to go about doing that. I guess we need to talk to infra about it? Do you know how we might clone the SVN repos themselves to test with?
>
> It looks like there are svn dumps at http://svn-master.apache.org/dump/ from 2 april 2011. You should be able to use those to setup a local subversion.
>
> -- Owen
>

Hey folks,

FYI I'm in the process of loading one of the SVN dumps onto a server here.

MAN is it slow. Going maybe 2 revisions/sec.... so I should have an SVN
replica in somewhere around a week to test with.

@Infra: I don't suppose it's possible to get a writable snapshot mounted
somehow? If I recall correctly, ZFS supports this and svn.apache.org runs
ZFS?
-Todd

On Fri, Apr 29, 2011 at 1:16 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:

> I can't do this at 2pm now.  Todd, I suspect you want more time to try out
> the svn/git test anyways.
>
> Let's shoot for next Wednesday at 2pm.  Ian should be back by then too.
>  Any objections?
>
> Cheers,
> Nige
>
> On Apr 29, 2011, at 11:36 AM, Owen O'Malley wrote:
>
> >
> > On Apr 28, 2011, at 11:24 PM, Todd Lipcon wrote:
> >
> >> Wasn't sure how to go about doing that. I guess we need to talk to infra
> about it? Do you know how we might clone the SVN repos themselves to test
> with?
> >
> > It looks like there are svn dumps at http://svn-master.apache.org/dump/from 2 april 2011. You should be able to use those to setup a local
> subversion.
> >
> > -- Owen
> >
>
>
--
Todd Lipcon
Software Engineer, Cloudera

You might want to look around at the options for imports more. I know
when I was doing git-svn clones it was doing something stupid like 1
rev/s until I found a --log-window setting. Not sure if that's from
the git layer or the svn layer but it made orders of magnitude
difference.

On Fri, May 6, 2011 at 4:04 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hey folks,
>
> FYI I'm in the process of loading one of the SVN dumps onto a server here.
>
> MAN is it slow. Going maybe 2 revisions/sec.... so I should have an SVN
> replica in somewhere around a week to test with.
>
> @Infra: I don't suppose it's possible to get a writable snapshot mounted
> somehow? If I recall correctly, ZFS supports this and svn.apache.org runs
> ZFS?
>
>
> -Todd
>
> On Fri, Apr 29, 2011 at 1:16 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:
>
>> I can't do this at 2pm now.  Todd, I suspect you want more time to try out
>> the svn/git test anyways.
>>
>> Let's shoot for next Wednesday at 2pm.  Ian should be back by then too.
>>  Any objections?
>>
>> Cheers,
>> Nige
>>
>> On Apr 29, 2011, at 11:36 AM, Owen O'Malley wrote:
>>
>> >
>> > On Apr 28, 2011, at 11:24 PM, Todd Lipcon wrote:
>> >
>> >> Wasn't sure how to go about doing that. I guess we need to talk to infra
>> about it? Do you know how we might clone the SVN repos themselves to test
>> with?
>> >
>> > It looks like there are svn dumps at http://svn-master.apache.org/dump/from 2 april 2011. You should be able to use those to setup a local
>> subversion.
>> >
>> > -- Owen
>> >
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Nigel,

I have committed federation merge into trunk. Thank you for waiting for it
to be done.

Regards,
Suresh
On 4/28/11 10:06 PM, "Nigel Daley" <[EMAIL PROTECTED]> wrote:

> As announced last week, I'm planning to do this at 2pm PDT tomorrow (Friday)
> April 29.  
>
> Suresh, when do you plan to commit HFS-1052?  That should be done first.
>
> Owen or Todd, did you want to follow Paul's advice:
>> If you're really wanting to make sure to keep the history in Git
>> intact my suggestion would be to setup a temporary svn server locally
>> and test our mirroring scripts against the commands you intend to run.
> If so, how much more time do you need?
>
> Cheers,
> Nige
>
> On Apr 20, 2011, at 9:42 PM, Nigel Daley wrote:
>
>> Owen, I'll admit I'm not familiar with all the git details/issues in your
>> proposal, but I think the layout change you propose is fine and seems to
>> solve the git issues with very minimal impact on the layout.
>>
>> Let's shoot for doing this next Friday, April 29 at 2pm PDT.  I'll update the
>> patch and send out a reminder about this later next week.
>>
>> Thanks,
>> Nige
>>
>> On Apr 20, 2011, at 8:00 AM, Owen O'Malley wrote:
>>
>>>
>>> On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:
>>>
>>>> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>>>
>>>>>
>>>>> I'm currently looking into how the git mirrors are setup in Apache-land.
>>>
>>> Uh, why isn't infra-dev on this thread?
>>>
>>> For those on infra-dev, the context is that Nigel is trying to merge
>>> together the source trees of the Hadoop sub-projects that were split apart 2
>>> years ago. So he is taking:
>>>
>>> prefix = http://svn.apache.org/repos/asf/hadoop/
>>>
>>> $prefix/common/trunk -> $prefix/trunk/common
>>> $prefix/hdfs/trunk -> $prefix/trunk/hdfs
>>> $prefix/mapreduce/trunk -> $prefix/trunk/mapreduce
>>>
>>> and play similar games with the rest of the branches and tags. For more
>>> details look at HADOOP-7106.
>>>
>>> From the project split, subversion was able to track the history across the
>>> subversion moves between projects, but not git.
>>>
>>> Four questions:
>>> 1. Is there anything we can do to minimize the history loss in git?
>>> 2. Are we going to be able to preserve our sha's or are they going to change
>>> again?
>>> 3. What changes do we need to make to the subversion notification file?
>>> 4. Are there any other changes that need to be coordinated?
>>>
>>> After considering it this morning, I believe that the least disruptive move
>>> is to leave common at the same url and merge hdfs and mapreduce back in:
>>>
>>> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
>>> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
>>> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
>>>
>>> This will preserve the hashes and history for common (and the 20 branches).
>>> We'll still need to play git voodoo to get git history for hdfs and
>>> mapreduce, but it is far better than starting a brand new git clone.
>>>
>>> -- Owen
>>>
>>>
>>
>

On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:

> As discussed in Jan/Feb, I'd like to coordinate a date for committing the re-organization of our svn layout: https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday April 21 at 11am PDT.

This is still premature.

Your patch is out of date with respect to the current branches. Please update it and upload a new version.

What changes do we want to make to the asf-authorization file? Clearly that needs to be decided before anything is done.

What changes are we making to the notifications? That also hasn't been discussed at all.

Again, has anyone talked to Jukka and the rest of the infrastructure-dev to see if there is anything that we need to do to minimize transition pain? Clearly, the notifications for git.apache.org need to be maintained, but I don't know if there are other actions that would help minimize the data loss.

Clearly any change where all of the git hashes change is massively disruptive. I suspect there is some magic you can do to keep more of the history, but I don't know the subversion-git bridge very well.

-- Owen