Sure, moving this discussion to the public list. It was only a private
discussion because I find people tend to notice messages in their inbox more
than on JIRA or a list ;-)

-Todd

On Wed, Nov 17, 2010 at 4:41 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:

> Can this message please be re-posted to the hdfs-dev list, so that any
> others who may be planning on work in this code also get a heads up?
> I don't see an advantage to this being a private discussion.
>
> Thanks,
> Jakob
>
> On Wed, Nov 17, 2010 at 4:14 PM, Sanjay Radia <[EMAIL PROTECTED]>
> wrote:
> >
> > On Nov 17, 2010, at 2:08 PM, Hairong Kuang wrote:
> >
> > Hi Todd,
> >
> > Here is the list of fsimage jiras that I am working on:
> >
> > HDFS-1481 // small change
> > HDFS-1458 // small change
> > HDFS-1496  // should be checked into 022, I still have no clue how to fix
> it
> > HDFS-1070  // this one has a simple idea, but turns out that it needs
> major
> > code re-organization. I am still struggling with how to make my code
> change
> > easier to review. But this will definitely collide with yours.
> >
> >  1070 is about changing the generation and processing of an fsimage.
> > It may not conflict with Todd's changes.
> > Todd if your refactoring is similar to the one outlined by Ivan in his
> jira
> > then it is good one for all to read.
> > Jakob has another related Jira.
> > +jakob.
> >
> > Hairong
> >
> > On 11/17/10 11:57 AM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:
> >
> > Hey all,
> >
> > As many of you know, I've been working on HDFS-1073 for a few months, and
> > we're hoping to get it in for 0.22 (OOM has given the OK to merge this
> into
> > branch and Sanjay has helpfully gotten some commitment from Yahoo QA to
> help
> > test it).
> >
> > It's been difficult recently to work on the patch as a lot of changes
> have
> > gone in around FSImage and FSEditLog (eg checksumming, parallel load,
> etc)
> > so the patch falls out of date quickly and is hard to resolve the changes
> > since my patch moves a lot of code. Ivan Kelly over at Yahoo Barcelona
> has
> > also been working on a refactor and I think running into a similar issue.
> >
> > So, I just wanted to ping everyone who has been working on this area of
> the
> > code to find out if there are any more remaining JIRAs that you're hoping
> to
> > commit in the next several weeks that will touch FSImage and FSEditLog.
> If
> > so, we should try to agree on an order in which to commit them so we
> don't
> > end up stomping on each other, and we can help review each other's work
> to
> > move quicker.
> >
> > If you can just reply with a list of any remaining uncommitted JIRAs that
> > touch FSImage/FSEditLog that would be great, then we can work from there
> on
> > how to proceed and minimize the amount of patch rebasing we all have to
> do.
> >
> > Thanks!
> > -Todd
> >
> >
>

--
Todd Lipcon
Software Engineer, Cloudera

I put up a sketch of a patch for 1473. It compiles and passes at least a
couple of the unit tests, but more work to be done on it. Would appreciate
comments from this crowd if the new classes seem like good divisions.

-Todd

On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <[EMAIL PROTECTED]>wrote:
>
>> You meant HDFS-1473 right? When do you expect that it could be done?
>>
>>
> I was working on HDFS-1473 this morning and could probably get a
> preliminary patch up today.. but when I realized that we have 4 parallel
> refactorings going on I paused my work. If we agree that HDFS-1473 will go
> in before the other ones I can finish it up, but if we're doing the same
> thing we should coordinate instead.
>
>
>> Hairong
>>
>> On 11/17/10 2:29 PM, "Eli Collins" <[EMAIL PROTECTED]> wrote:
>>
>> > Agree we should do the refactoring before the features, we should also
>> > add the tests with the refactoring to show they don't break things.
>> >
>> > On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>> >> OK, looks like there is still quite a bit more in motion here, thanks
>> for
>> >> the list, Hairong.
>> >> How would everyone feel about setting aside those improvements/fixes,
>> doing
>> >> the refactor first, and then going back to them? It should help with
>> testing
>> >> and reduce conflicts between the various ongoing projects.
>> >> Maybe we can discuss quickly on this thread: what are some small
>> refactors
>> >> we can start with to help separate these things? Ivan's project looks
>> really
>> >> good but it seems like it is a pretty large change. If we can break it
>> up
>> >> into a couple smaller patches we can commit as we go.
>> >> -Todd
>> >> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <[EMAIL PROTECTED]>
>> wrote:
>> >>>
>> >>> Hi Todd,
>> >>>
>> >>> Here is the list of fsimage jiras that I am working on:
>> >>>
>> >>> HDFS-1481 // small change
>> >>> HDFS-1458 // small change
>> >>> HDFS-1496  // should be checked into 022, I still have no clue how to
>> fix
>> >>> it
>> >>> HDFS-1070  // this one has a simple idea, but turns out that it needs
>> >>> major code re-organization. I am still struggling with how to make my
>> code
>> >>> change easier to review. But this will definitely collide with yours.
>> >>>
>> >>> Hairong
>> >>>
>> >>> On 11/17/10 11:57 AM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:
>> >>>
>> >>> Hey all,
>> >>>
>> >>> As many of you know, I've been working on HDFS-1073 for a few months,
>> and
>> >>> we're hoping to get it in for 0.22 (OOM has given the OK to merge this
>> into
>> >>> branch and Sanjay has helpfully gotten some commitment from Yahoo QA
>> to help
>> >>> test it).
>> >>>
>> >>> It's been difficult recently to work on the patch as a lot of changes
>> have
>> >>> gone in around FSImage and FSEditLog (eg checksumming, parallel load,
>> etc)
>> >>> so the patch falls out of date quickly and is hard to resolve the
>> changes
>> >>> since my patch moves a lot of code. Ivan Kelly over at Yahoo Barcelona
>> has
>> >>> also been working on a refactor and I think running into a similar
>> issue.
>> >>>
>> >>> So, I just wanted to ping everyone who has been working on this area
>> of
>> >>> the code to find out if there are any more remaining JIRAs that you're
>> >>> hoping to commit in the next several weeks that will touch FSImage and
>> >>> FSEditLog. If so, we should try to agree on an order in which to
>> commit them
>> >>> so we don't end up stomping on each other, and we can help review each
>> >>> other's work to move quicker.
>> >>>
>> >>> If you can just reply with a list of any remaining uncommitted JIRAs
>> that
>> >>> touch FSImage/FSEditLog that would be great, then we can work from
>> there on
>> >>> how to proceed and minimize the amount of patch rebasing we all have
>> to do.
>> >>>
>> >>> Thanks!
>> >>> -Todd
>> >>
>> >>
>> >>
>> >> --
>> >> Todd Lipcon
>> >> Software Engineer, Cloudera

Todd Lipcon
Software Engineer, Cloudera

This patch looks really good. It shouldn't affect our refactoring too much and will actually make things a lot easier for the next part (breaking the circular dependency between namesystem and fsimage).

-Ivan

On 18 Nov 2010, at 02:27, Todd Lipcon wrote:

I put up a sketch of a patch for 1473. It compiles and passes at least a couple of the unit tests, but more work to be done on it. Would appreciate comments from this crowd if the new classes seem like good divisions.

-Todd

On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
You meant HDFS-1473 right? When do you expect that it could be done?
I was working on HDFS-1473 this morning and could probably get a preliminary patch up today.. but when I realized that we have 4 parallel refactorings going on I paused my work. If we agree that HDFS-1473 will go in before the other ones I can finish it up, but if we're doing the same thing we should coordinate instead.

Hairong

On 11/17/10 2:29 PM, "Eli Collins" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:

> Agree we should do the refactoring before the features, we should also
> add the tests with the refactoring to show they don't break things.
>
> On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> OK, looks like there is still quite a bit more in motion here, thanks for
>> the list, Hairong.
>> How would everyone feel about setting aside those improvements/fixes, doing
>> the refactor first, and then going back to them? It should help with testing
>> and reduce conflicts between the various ongoing projects.
>> Maybe we can discuss quickly on this thread: what are some small refactors
>> we can start with to help separate these things? Ivan's project looks really
>> good but it seems like it is a pretty large change. If we can break it up
>> into a couple smaller patches we can commit as we go.
>> -Todd
>> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>
>>> Hi Todd,
>>>
>>> Here is the list of fsimage jiras that I am working on:
>>>
>>> HDFS-1481 // small change
>>> HDFS-1458 // small change
>>> HDFS-1496  // should be checked into 022, I still have no clue how to fix
>>> it
>>> HDFS-1070  // this one has a simple idea, but turns out that it needs
>>> major code re-organization. I am still struggling with how to make my code
>>> change easier to review. But this will definitely collide with yours.
>>>
>>> Hairong
>>>
>>> On 11/17/10 11:57 AM, "Todd Lipcon" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>
>>> Hey all,
>>>
>>> As many of you know, I've been working on HDFS-1073 for a few months, and
>>> we're hoping to get it in for 0.22 (OOM has given the OK to merge this into
>>> branch and Sanjay has helpfully gotten some commitment from Yahoo QA to help
>>> test it).
>>>
>>> It's been difficult recently to work on the patch as a lot of changes have
>>> gone in around FSImage and FSEditLog (eg checksumming, parallel load, etc)
>>> so the patch falls out of date quickly and is hard to resolve the changes
>>> since my patch moves a lot of code. Ivan Kelly over at Yahoo Barcelona has
>>> also been working on a refactor and I think running into a similar issue.
>>>
>>> So, I just wanted to ping everyone who has been working on this area of
>>> the code to find out if there are any more remaining JIRAs that you're
>>> hoping to commit in the next several weeks that will touch FSImage and
>>> FSEditLog. If so, we should try to agree on an order in which to commit them
>>> so we don't end up stomping on each other, and we can help review each
>>> other's work to move quicker.
>>>
>>> If you can just reply with a list of any remaining uncommitted JIRAs that
>>> touch FSImage/FSEditLog that would be great, then we can work from there on
>>> how to proceed and minimize the amount of patch rebasing we all have to do.
Todd Lipcon
Software Engineer, Cloudera

Todd Lipcon
Software Engineer, Cloudera

Glancing at 1473, it seems reasonable, but of course should get a
thorough review.  HDFS-1465 will impact and essentially replace all the
saving/loading code, so whether the it does so with the code's current
state or with 1473's refactoring, it doesn't matter much.  1473 will
probably make it easier.

It would be more useful if 1465 went in before 1070, because it would
save Hairong work, but I believe she's already started and I just don't
have the bandwidth to do it at the moment, but I'm hoping that it will
be the next thing I work on.  The end result for the image, when we're
all done, will hopefully be refactored, unified and efficient
loading/saving code.

It should be noted that HDFS-1448, and the de-duplication JIRA that's
been agreed to as its sequel, will also impact on all of the edits work.
-jg

Ivan Kelly wrote:
> This patch looks really good. It shouldn't affect our refactoring too much and will actually make things a lot easier for the next part (breaking the circular dependency between namesystem and fsimage).
>
> -Ivan
>
> On 18 Nov 2010, at 02:27, Todd Lipcon wrote:
>
> I put up a sketch of a patch for 1473. It compiles and passes at least a couple of the unit tests, but more work to be done on it. Would appreciate comments from this crowd if the new classes seem like good divisions.
>
> -Todd
>
> On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> You meant HDFS-1473 right? When do you expect that it could be done?
>
>
> I was working on HDFS-1473 this morning and could probably get a preliminary patch up today.. but when I realized that we have 4 parallel refactorings going on I paused my work. If we agree that HDFS-1473 will go in before the other ones I can finish it up, but if we're doing the same thing we should coordinate instead.
>
> Hairong
>
> On 11/17/10 2:29 PM, "Eli Collins" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>
>> Agree we should do the refactoring before the features, we should also
>> add the tests with the refactoring to show they don't break things.
>>
>> On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>> OK, looks like there is still quite a bit more in motion here, thanks for
>>> the list, Hairong.
>>> How would everyone feel about setting aside those improvements/fixes, doing
>>> the refactor first, and then going back to them? It should help with testing
>>> and reduce conflicts between the various ongoing projects.
>>> Maybe we can discuss quickly on this thread: what are some small refactors
>>> we can start with to help separate these things? Ivan's project looks really
>>> good but it seems like it is a pretty large change. If we can break it up
>>> into a couple smaller patches we can commit as we go.
>>> -Todd
>>> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>> Hi Todd,
>>>>
>>>> Here is the list of fsimage jiras that I am working on:
>>>>
>>>> HDFS-1481 // small change
>>>> HDFS-1458 // small change
>>>> HDFS-1496  // should be checked into 022, I still have no clue how to fix
>>>> it
>>>> HDFS-1070  // this one has a simple idea, but turns out that it needs
>>>> major code re-organization. I am still struggling with how to make my code
>>>> change easier to review. But this will definitely collide with yours.
>>>>
>>>> Hairong
>>>>
>>>> On 11/17/10 11:57 AM, "Todd Lipcon" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>>
>>>> Hey all,
>>>>
>>>> As many of you know, I've been working on HDFS-1073 for a few months, and
>>>> we're hoping to get it in for 0.22 (OOM has given the OK to merge this into
>>>> branch and Sanjay has helpfully gotten some commitment from Yahoo QA to help
>>>> test it).
>>>>
>>>> It's been difficult recently to work on the patch as a lot of changes have

Hi all,

There's a patch up for 1473 now. If we can get a +1 we can commit this and
move on to the next steps in some of these projects.

Thanks
-Todd

On Thu, Nov 18, 2010 at 12:42 AM, Ivan Kelly <[EMAIL PROTECTED]> wrote:

> This patch looks really good. It shouldn't affect our refactoring too much
> and will actually make things a lot easier for the next part (breaking the
> circular dependency between namesystem and fsimage).
>
> -Ivan
>
>
> On 18 Nov 2010, at 02:27, Todd Lipcon wrote:
>
> I put up a sketch of a patch for 1473. It compiles and passes at least a
> couple of the unit tests, but more work to be done on it. Would appreciate
> comments from this crowd if the new classes seem like good divisions.
>
> -Todd
>
> On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <[EMAIL PROTECTED]>wrote:
>>
>>> You meant HDFS-1473 right? When do you expect that it could be done?
>>>
>>>
>> I was working on HDFS-1473 this morning and could probably get a
>> preliminary patch up today.. but when I realized that we have 4 parallel
>> refactorings going on I paused my work. If we agree that HDFS-1473 will go
>> in before the other ones I can finish it up, but if we're doing the same
>> thing we should coordinate instead.
>>
>>
>>> Hairong
>>>
>>> On 11/17/10 2:29 PM, "Eli Collins" <[EMAIL PROTECTED]> wrote:
>>>
>>> > Agree we should do the refactoring before the features, we should also
>>> > add the tests with the refactoring to show they don't break things.
>>> >
>>> > On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <[EMAIL PROTECTED]>
>>> wrote:
>>> >> OK, looks like there is still quite a bit more in motion here, thanks
>>> for
>>> >> the list, Hairong.
>>> >> How would everyone feel about setting aside those improvements/fixes,
>>> doing
>>> >> the refactor first, and then going back to them? It should help with
>>> testing
>>> >> and reduce conflicts between the various ongoing projects.
>>> >> Maybe we can discuss quickly on this thread: what are some small
>>> refactors
>>> >> we can start with to help separate these things? Ivan's project looks
>>> really
>>> >> good but it seems like it is a pretty large change. If we can break it
>>> up
>>> >> into a couple smaller patches we can commit as we go.
>>> >> -Todd
>>> >> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <[EMAIL PROTECTED]>
>>> wrote:
>>> >>>
>>> >>> Hi Todd,
>>> >>>
>>> >>> Here is the list of fsimage jiras that I am working on:
>>> >>>
>>> >>> HDFS-1481 // small change
>>> >>> HDFS-1458 // small change
>>> >>> HDFS-1496  // should be checked into 022, I still have no clue how to
>>> fix
>>> >>> it
>>> >>> HDFS-1070  // this one has a simple idea, but turns out that it needs
>>> >>> major code re-organization. I am still struggling with how to make my
>>> code
>>> >>> change easier to review. But this will definitely collide with yours.
>>> >>>
>>> >>> Hairong
>>> >>>
>>> >>> On 11/17/10 11:57 AM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:
>>> >>>
>>> >>> Hey all,
>>> >>>
>>> >>> As many of you know, I've been working on HDFS-1073 for a few months,
>>> and
>>> >>> we're hoping to get it in for 0.22 (OOM has given the OK to merge
>>> this into
>>> >>> branch and Sanjay has helpfully gotten some commitment from Yahoo QA
>>> to help
>>> >>> test it).
>>> >>>
>>> >>> It's been difficult recently to work on the patch as a lot of changes
>>> have
>>> >>> gone in around FSImage and FSEditLog (eg checksumming, parallel load,
>>> etc)
>>> >>> so the patch falls out of date quickly and is hard to resolve the
>>> changes
>>> >>> since my patch moves a lot of code. Ivan Kelly over at Yahoo
>>> Barcelona has
>>> >>> also been working on a refactor and I think running into a similar
>>> issue.
>>> >>>
>>> >>> So, I just wanted to ping everyone who has been working on this area
>>> of
>>> >>> the code to find out if there are any more remaining JIRAs that
>>> you're
>>> >>> hoping to commit in the next several weeks that will touch FSImage
Todd Lipcon
Software Engineer, Cloudera

Hey Todd, Ivan,

I've committed the patch for 1473.

Thanks,
Eli

On Wed, Nov 24, 2010 at 2:25 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> There's a patch up for 1473 now. If we can get a +1 we can commit this and
> move on to the next steps in some of these projects.
>
> Thanks
> -Todd
>
> On Thu, Nov 18, 2010 at 12:42 AM, Ivan Kelly <[EMAIL PROTECTED]> wrote:
>
>> This patch looks really good. It shouldn't affect our refactoring too much
>> and will actually make things a lot easier for the next part (breaking the
>> circular dependency between namesystem and fsimage).
>>
>> -Ivan
>>
>>
>> On 18 Nov 2010, at 02:27, Todd Lipcon wrote:
>>
>> I put up a sketch of a patch for 1473. It compiles and passes at least a
>> couple of the unit tests, but more work to be done on it. Would appreciate
>> comments from this crowd if the new classes seem like good divisions.
>>
>> -Todd
>>
>> On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>
>>> On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <[EMAIL PROTECTED]>wrote:
>>>
>>>> You meant HDFS-1473 right? When do you expect that it could be done?
>>>>
>>>>
>>> I was working on HDFS-1473 this morning and could probably get a
>>> preliminary patch up today.. but when I realized that we have 4 parallel
>>> refactorings going on I paused my work. If we agree that HDFS-1473 will go
>>> in before the other ones I can finish it up, but if we're doing the same
>>> thing we should coordinate instead.
>>>
>>>
>>>> Hairong
>>>>
>>>> On 11/17/10 2:29 PM, "Eli Collins" <[EMAIL PROTECTED]> wrote:
>>>>
>>>> > Agree we should do the refactoring before the features, we should also
>>>> > add the tests with the refactoring to show they don't break things.
>>>> >
>>>> > On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >> OK, looks like there is still quite a bit more in motion here, thanks
>>>> for
>>>> >> the list, Hairong.
>>>> >> How would everyone feel about setting aside those improvements/fixes,
>>>> doing
>>>> >> the refactor first, and then going back to them? It should help with
>>>> testing
>>>> >> and reduce conflicts between the various ongoing projects.
>>>> >> Maybe we can discuss quickly on this thread: what are some small
>>>> refactors
>>>> >> we can start with to help separate these things? Ivan's project looks
>>>> really
>>>> >> good but it seems like it is a pretty large change. If we can break it
>>>> up
>>>> >> into a couple smaller patches we can commit as we go.
>>>> >> -Todd
>>>> >> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >>>
>>>> >>> Hi Todd,
>>>> >>>
>>>> >>> Here is the list of fsimage jiras that I am working on:
>>>> >>>
>>>> >>> HDFS-1481 // small change
>>>> >>> HDFS-1458 // small change
>>>> >>> HDFS-1496  // should be checked into 022, I still have no clue how to
>>>> fix
>>>> >>> it
>>>> >>> HDFS-1070  // this one has a simple idea, but turns out that it needs
>>>> >>> major code re-organization. I am still struggling with how to make my
>>>> code
>>>> >>> change easier to review. But this will definitely collide with yours.
>>>> >>>
>>>> >>> Hairong
>>>> >>>
>>>> >>> On 11/17/10 11:57 AM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:
>>>> >>>
>>>> >>> Hey all,
>>>> >>>
>>>> >>> As many of you know, I've been working on HDFS-1073 for a few months,
>>>> and
>>>> >>> we're hoping to get it in for 0.22 (OOM has given the OK to merge
>>>> this into
>>>> >>> branch and Sanjay has helpfully gotten some commitment from Yahoo QA
>>>> to help
>>>> >>> test it).
>>>> >>>
>>>> >>> It's been difficult recently to work on the patch as a lot of changes
>>>> have
>>>> >>> gone in around FSImage and FSEditLog (eg checksumming, parallel load,
>>>> etc)
>>>> >>> so the patch falls out of date quickly and is hard to resolve the
>>>> changes
>>>> >>> since my patch moves a lot of code. Ivan Kelly over at Yahoo
>>>> Barcelona has
>>>> >>> also been working on a refactor and I think running into a similar