Hey gang,

We've had some discussion on what to do with regard to MR1 when MR2
gets merged in, and wanted to give you a heads up.  By MR1 I mean the
current MR implementation that uses the JobTracker, TaskTracker, etc.
On this thread (http://search-hadoop.com/m/GJliJ1uwjXu) on
mapreduce-dev@ we came to consensus that it makes sense to remove the
MR1 code from trunk (and the 23 release) and only support the MR2
implementation in 23 and going forward. In short, there are currently
three separate MR implementations and we'd like to only maintain two
(MR1 in stable, MR2 in trunk/23).  Note! MR2 supports the current job
API - users don't need to rewrite their jobs to run on MR2 - this is
about the MR *implementation* not job compatibility.  Note that the
move to MR2 will affect some APIs (eg metrics, contrib projects that
only work against MR1, etc). The current MR1 implementation will of
course remain supported in the current stable releases.

Rationale: there's a lot of cost but little gain to maintaining three
MR implementations. Getting the MR1 code in trunk in shape so that it
is comparable in reliability/performance/features to the stable MR1
code is a lot of work. Eg security is still not supported by MR1 in
trunk, doesn't look like that's getting closed out, it hasn't been
tested at scale, etc. And it is unlikely that anyone will volunteer to
do this work given that we are moving to MR2. Ie if you want to use
MR1 we'd recommend the stable release, and if you're using 23 we'd
recommend MR2, therefore, given that we wouldn't recommend anyone use
MR1 in trunk/23 it doesn't make sense to ship it.

The current plan is to remove the MR1 code from trunk after merging in MR-279.

Thanks,
Eli

On Jul 25, 2011, at 11:59 AM, Eli Collins wrote:
> Note! MR2 supports the current job
> API - users don't need to rewrite their jobs to run on MR2 - this is
> about the MR *implementation* not job compatibility.  Note that the
> move to MR2 will affect some APIs (eg metrics, contrib projects that
> only work against MR1, etc).

Don't forget that this completely breaks operations, as in our entire world is about to get blown apart and it will take us quite a while to recover.

> The current MR1 implementation will of
> course remain supported in the current stable releases.
>
> Rationale: there's a lot of cost but little gain to maintaining three
> MR implementations.

What are the three implementations?  I only see two?

On Mon, Jul 25, 2011 at 3:17 PM, Allen Wittenauer <[EMAIL PROTECTED]> wrote:
>
> On Jul 25, 2011, at 11:59 AM, Eli Collins wrote:
>> Note! MR2 supports the current job
>> API - users don't need to rewrite their jobs to run on MR2 - this is
>> about the MR *implementation* not job compatibility.  Note that the
>> move to MR2 will affect some APIs (eg metrics, contrib projects that
>> only work against MR1, etc).
>
>        Don't forget that this completely breaks operations, as in our entire world is about to get blown apart and it will take us quite a while to recover.
>

Yes, moving operations from MR1 to MR2 will require some work.

>> The current MR1 implementation will of
>> course remain supported in the current stable releases.
>>
>> Rationale: there's a lot of cost but little gain to maintaining three
>> MR implementations.
>
>        What are the three implementations?  I only see two?
>

1. MR1 in stable
2. MR1 in trunk
3. MR2 in trunk

Given that we only recommend #1 and #3 it doesn't make sense to support #2.

Thanks,
Eli

On Jul 25, 2011, at 11:59 AM, Eli Collins wrote:

>
> The current plan is to remove the MR1 code from trunk after merging in MR-279.
>

One issue is that lots of MR 'unit tests' rely on MiniMRCluster which exposes very minute aspects of the JT/TT - something we can't support in MR2. Given that we might not have the time to port all the tests we might want to keep MR1 but delete the scripts in bin/.

Thoughts?

Arun

On Mon, Jul 25, 2011 at 5:23 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
> On Jul 25, 2011, at 11:59 AM, Eli Collins wrote:
>
>>
>> The current plan is to remove the MR1 code from trunk after merging in MR-279.
>>
>
> One issue is that lots of MR 'unit tests' rely on MiniMRCluster which exposes very minute aspects of the JT/TT - something we can't support in MR2. Given that we might not have the time to port all the tests we might want to keep MR1 but delete the scripts in bin/.
>

How much effort would it take to port them?  If we could get it to a
state that's easily parallelizable it would be a good hackathon.
(ditto for moving to FileContext btw).

Keeping the infrastructure but removing bin would work too. I wonder
though if we've lost much coverage if the tests depend on the JT/TT.

Thanks,
Eli

On 26/07/11 01:23, Arun C Murthy wrote:
>
> On Jul 25, 2011, at 11:59 AM, Eli Collins wrote:
>
>>
>> The current plan is to remove the MR1 code from trunk after merging in MR-279.
>>
>
> One issue is that lots of MR 'unit tests' rely on MiniMRCluster which exposes very minute aspects of the JT/TT - something we can't support in MR2. Given that we might not have the time to port all the tests we might want to keep MR1 but delete the scripts in bin/.
>

Why does the MiniMR cluster go away in MR2?

I quite like it in my own code as I can set it up for a test run, push
some tests through it and then tear it down. It's got weaknesses

-doesn't shut down very well, tends to hold locks on build/test/metadata
or something. This doesn't show up in Hadoop JUnit as every Test class
runs in its own process, but in my functional tests I do bring up the
cluster in a separate process and push out work to it, and I need to
leave the cluster live for the entire test run, as per-test-class
doesn't work.

-does have hardcoded assumptions that there is a "./build/test" subdir
under which it should create a new dir (not configurable)

These are minor issues. The key thing is that MiniDFS cluster and MiniMR
cluster let you bring up a short lived MR cluster without any
installation overhead and run tests against it. Even if the
hadoop-mapreduce code can move away from it, it's still invaluable
downstream

On Tue, Jul 26, 2011 at 4:52 PM, Steve Loughran <[EMAIL PROTECTED]> wrote:
>
>
> Why does the MiniMR cluster go away in MR2?
>

As Arun pointed out, MiniMR has lot of JT and TT bits ingrained in it which
makes it almost unusable in MR2. I wish tests would have used MiniMR via a
interface then the implementation could be changed to reflect MR2 runtime.

>
>
> These are minor issues. The key thing is that MiniDFS cluster and MiniMR
> cluster let you bring up a short lived MR cluster without any installation
> overhead and run tests against it. Even if the hadoop-mapreduce code can
> move away from it, it's still invaluable downstream
>
>
Absolutely, infact MR2 has got similar things to write effective tests
easily :
- MiniYarnCluster -  includes RM and NM
- MiniMRYarnCluster (extends MiniYarnCluster) - includes AM and MR history
server

These follow Service lifecycles, so much easier to extend and work with.

Thanks
Sharad

On 27/07/11 06:38, Sharad Agarwal wrote:
> On Tue, Jul 26, 2011 at 4:52 PM, Steve Loughran<[EMAIL PROTECTED]>  wrote:
>>
>>
>> Why does the MiniMR cluster go away in MR2?
>>
>
> As Arun pointed out, MiniMR has lot of JT and TT bits ingrained in it which
> makes it almost unusable in MR2. I wish tests would have used MiniMR via a
> interface then the implementation could be changed to reflect MR2 runtime.
>
>>
>>
>> These are minor issues. The key thing is that MiniDFS cluster and MiniMR
>> cluster let you bring up a short lived MR cluster without any installation
>> overhead and run tests against it. Even if the hadoop-mapreduce code can
>> move away from it, it's still invaluable downstream
>>
>>
> Absolutely, infact MR2 has got similar things to write effective tests
> easily :
> - MiniYarnCluster -  includes RM and NM
> - MiniMRYarnCluster (extends MiniYarnCluster) - includes AM and MR history
> server
>
> These follow Service lifecycles, so much easier to extend and work with.
>
> Thanks
> Sharad
>

As long as there is an in-vm mini cluster for testing I'm happy. I am
building up a list of things to fix in MiniDFS cluster, which is going
to stay as is, right?

On Wed, Jul 27, 2011 at 3:09 PM, Steve Loughran <[EMAIL PROTECTED]> wrote:

As long as there is an in-vm mini cluster for testing I'm happy. I am
> building up a list of things to fix in MiniDFS cluster, which is going to
> stay as is, right?
>

Right. MR2 doesn't change anything on HDFS side.

Sharad

On Tue, Jul 26, 2011 at 10:38 PM, Sharad Agarwal
<[EMAIL PROTECTED]> wrote:
> On Tue, Jul 26, 2011 at 4:52 PM, Steve Loughran <[EMAIL PROTECTED]> wrote:
>>
>>
>> Why does the MiniMR cluster go away in MR2?
>>
>
> As Arun pointed out, MiniMR has lot of JT and TT bits ingrained in it which
> makes it almost unusable in MR2. I wish tests would have used MiniMR via a
> interface then the implementation could be changed to reflect MR2 runtime.
>
>>
>>
>> These are minor issues. The key thing is that MiniDFS cluster and MiniMR
>> cluster let you bring up a short lived MR cluster without any installation
>> overhead and run tests against it. Even if the hadoop-mapreduce code can
>> move away from it, it's still invaluable downstream
>>
>>
> Absolutely, infact MR2 has got similar things to write effective tests
> easily :
> - MiniYarnCluster -  includes RM and NM
> - MiniMRYarnCluster (extends MiniYarnCluster) - includes AM and MR history
> server
>
> These follow Service lifecycles, so much easier to extend and work with.
>

Is the plan to port all the current MR tests to the new Mini*Cluster classes?

Thanks,
Eli

On Jul 27, 2011, at 9:41 AM, Eli Collins wrote:
> Is the plan to port all the current MR tests to the new Mini*Cluster classes?

I don't think a straight port is the right way to go - we should stop relying on the Mini*Clusters for every single test and go the route of unit/mock tests. The Mini*Clusters add too much overhead and aren't the right way forward, IMO.

IAC, we can separate these discussions. So, for now, I propose we just remove the bin/* scripts for the purposes of this discussion. Makes sense?

thanks,
Arun

On Wed, Jul 27, 2011 at 9:53 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>
> On Jul 27, 2011, at 9:41 AM, Eli Collins wrote:
>> Is the plan to port all the current MR tests to the new Mini*Cluster classes?
>
> I don't think a straight port is the right way to go - we should stop relying on the Mini*Clusters for every single test and go the route of unit/mock tests. The Mini*Clusters add too much overhead and aren't the right way forward, IMO.
>

Agree, a lot of what the mini mr tests are trying to cover would be
better written (and 10x faster) as unit tests.  If MR2 has the
equivalent coverage in unit tests then we should then we should remove
the old MiniMR tests.

> IAC, we can separate these discussions. So, for now, I propose we just remove the bin/* scripts for the purposes of this discussion. Makes sense?

Sure, filed MAPREDUCE-2736. We can discuss specifics there.

Thanks,
Eli