Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # dev - [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')


+
Alejandro Abdelnur 2012-12-05, 17:24
Copy link to this message
-
Re: [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')
Colin McCabe 2012-12-05, 18:35
I think this is a good idea, for a few different reasons.

* We all know Java, so this code will be readable by all.
* It will benefit the wider community of Maven users, not just our project.
* It gets rid of the shell dependency.  The shell dependency is
problematic because Windows doesn't support most UNIX shell code, and
also because different Linux distributions have different default
shells.  The /bin/dash versus /bin/bash issue has caused problems in
the past.

Is there any way this code could be used in branch-1 to solve some of
the issues there?  I know the protoc code, at least, would be useful
there.

regards,
Colin
On Wed, Dec 5, 2012 at 9:24 AM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
> On the '[VOTE] introduce Python as build-time and run-time dependency for
> Hadoop and throughout Hadoop stack' thread the vote for using Maven plugins
> for the build had an overwhelming acceptance (several +1s, a few 0s and no
> -1s).
>
> Current tasks of the build for being 'pluginized' are:
>
> 1* cmake (HADOOP-8887)
> 2* protoc(HADOOP-9117)
> 3* saveVersion (HADOOP-8924) (but this one has been -1)
>
> I've tested #2 and #3 in Windows an the worked as expected. Regarding #1,
> if cmake is supported (and used for hadoop) in Window, i believe it should
> work with minimal changes.
>
> As I mentioned in HADOOP-8924, "Writing Maven plugins is more complex that
> writing scripts, I don't dispute this. The main motivation for using Maven
> plugins is to keep things in the POM declarative and to hide (if necessary)
> different handlings for different platforms."
>
> In order to enable the use of custom plugins for the Hadoop build, we need
> to have such plugins available in a Maven repo for Maven to download and
> use them. Note that they cannot be part of the same build that is building
> Hadoop itself.
>
> While we could argue this plugins are general purpose and they should be
> developed in a different standalone project, doing this would definitely
> slow down when we can use them in Hadoop.
>
> Because of this, I propose we create a Hadoop subproject 'hadoop-build' to
> host custom Maven plugins (and custom Ant tasks for branch-1) for Hadoop.
>
> Being a sub-project will allow us to do independent releases (from Hadoop
> releases) of plugins  we need for Hadoop build. Plus, use them immediately
> in the Hadoop build.
>
> Eventually, if the Maven community picks up some of these plugins we could
> simply remove them from our hadoop-build project and change Hadoop POMs to
> use the new ones.
>
> Looking forward to hear what others think.
>
> Thanks.
>
> --
> Alejandro
+
Alejandro Abdelnur 2012-12-05, 18:41
+
Doug Cutting 2012-12-08, 00:20
+
Radim Kolar 2012-12-08, 01:31
+
Colin McCabe 2012-12-10, 18:50
+
Colin McCabe 2012-12-10, 19:33
+
Radim Kolar 2012-12-12, 03:16
+
Colin McCabe 2012-12-12, 18:22
+
Alejandro Abdelnur 2012-12-12, 18:20
+
Doug Cutting 2012-12-12, 18:46