Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - Re: Changes to FSImage/FSEditLog


Copy link to this message
-
Re: Changes to FSImage/FSEditLog
Ivan Kelly 2010-11-18, 08:42
This patch looks really good. It shouldn't affect our refactoring too much and will actually make things a lot easier for the next part (breaking the circular dependency between namesystem and fsimage).

-Ivan

On 18 Nov 2010, at 02:27, Todd Lipcon wrote:

I put up a sketch of a patch for 1473. It compiles and passes at least a couple of the unit tests, but more work to be done on it. Would appreciate comments from this crowd if the new classes seem like good divisions.

-Todd

On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
You meant HDFS-1473 right? When do you expect that it could be done?
I was working on HDFS-1473 this morning and could probably get a preliminary patch up today.. but when I realized that we have 4 parallel refactorings going on I paused my work. If we agree that HDFS-1473 will go in before the other ones I can finish it up, but if we're doing the same thing we should coordinate instead.

Hairong

On 11/17/10 2:29 PM, "Eli Collins" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:

> Agree we should do the refactoring before the features, we should also
> add the tests with the refactoring to show they don't break things.
>
> On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> OK, looks like there is still quite a bit more in motion here, thanks for
>> the list, Hairong.
>> How would everyone feel about setting aside those improvements/fixes, doing
>> the refactor first, and then going back to them? It should help with testing
>> and reduce conflicts between the various ongoing projects.
>> Maybe we can discuss quickly on this thread: what are some small refactors
>> we can start with to help separate these things? Ivan's project looks really
>> good but it seems like it is a pretty large change. If we can break it up
>> into a couple smaller patches we can commit as we go.
>> -Todd
>> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>
>>> Hi Todd,
>>>
>>> Here is the list of fsimage jiras that I am working on:
>>>
>>> HDFS-1481 // small change
>>> HDFS-1458 // small change
>>> HDFS-1496  // should be checked into 022, I still have no clue how to fix
>>> it
>>> HDFS-1070  // this one has a simple idea, but turns out that it needs
>>> major code re-organization. I am still struggling with how to make my code
>>> change easier to review. But this will definitely collide with yours.
>>>
>>> Hairong
>>>
>>> On 11/17/10 11:57 AM, "Todd Lipcon" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>
>>> Hey all,
>>>
>>> As many of you know, I've been working on HDFS-1073 for a few months, and
>>> we're hoping to get it in for 0.22 (OOM has given the OK to merge this into
>>> branch and Sanjay has helpfully gotten some commitment from Yahoo QA to help
>>> test it).
>>>
>>> It's been difficult recently to work on the patch as a lot of changes have
>>> gone in around FSImage and FSEditLog (eg checksumming, parallel load, etc)
>>> so the patch falls out of date quickly and is hard to resolve the changes
>>> since my patch moves a lot of code. Ivan Kelly over at Yahoo Barcelona has
>>> also been working on a refactor and I think running into a similar issue.
>>>
>>> So, I just wanted to ping everyone who has been working on this area of
>>> the code to find out if there are any more remaining JIRAs that you're
>>> hoping to commit in the next several weeks that will touch FSImage and
>>> FSEditLog. If so, we should try to agree on an order in which to commit them
>>> so we don't end up stomping on each other, and we can help review each
>>> other's work to move quicker.
>>>
>>> If you can just reply with a list of any remaining uncommitted JIRAs that
>>> touch FSImage/FSEditLog that would be great, then we can work from there on
>>> how to proceed and minimize the amount of patch rebasing we all have to do.
Todd Lipcon
Software Engineer, Cloudera

Todd Lipcon
Software Engineer, Cloudera