Matt Foley 2012-11-21, 19:15
Radim Kolar 2012-11-23, 23:40
Matt Foley 2012-11-24, 20:13
Radim Kolar 2012-11-24, 21:26
Konstantin Boudnik 2012-11-24, 22:03
-Re: [PROPOSAL] introduce Python as build-time and run-time dependency for Hadoop and throughout Hadoop stack
Alejandro Abdelnur 2012-11-21, 19:25
We already require java/mvn/protoc/cmake/forrest (forrest is hopefully on
its way out with the move of docs to APT)
Why not do a maven-plugin to do that?
Colin already has something to simplify all the cmake calls from the builds
using a maven-plugin (https://issues.apache.org/jira/browse/HADOOP-8887)
We could do the same with protoc, thus simplifying the POMs.
The saveVersion.sh seems like another prime candidate for a maven plugin,
and in this case it would not require external tools.
Does this make sense?
On Wed, Nov 21, 2012 at 11:15 AM, Matt Foley <[EMAIL PROTECTED]> wrote:
> This discussion started in
> , where it was proposed to replace the build-time utility "saveVersion.sh"
> with a python script. This would require Python as a build-time
> dependency. Here's the background:
> Those of us involved in the branch-1-win port of Hadoop to Windows without
> use of Cygwin, have faced the issue of frequent use of shell scripts
> throughout the system, both in build time (eg, the utility
> and run time (config files like "hadoop-env.sh" and the start/stop scripts
> in "bin/*" ). Similar usages exist throughout the Hadoop stack, in all
> The vast majority of these shell scripts do not do anything platform
> specific; they can be expressed in a posix-conforming way. Therefore, it
> seems to us that it makes sense to start using a cross-platform scripting
> language, such as python, in place of shell for these purposes. For those
> rare occasions where platform-specific functionality really is needed,
> python also supports quite a lot of platform-specific functionality on both
> Linux and Windows; but where that is inadequate, one could still
> conditionally invoke a platform-specific module written in shell (for
> Linux/*nix) or powershell or bat (for Windows).
> The primary motive for moving to a cross-platform scripting language is
> maintainability. The alternative would be to maintain two complete suites
> of scripts, one for Linux and one for Windows (and perhaps others in the
> future). We want to avoid the need to update dual modules in two different
> languages when functionality changes, especially given that many Linux
> developers are not familiar with powershell or bat, and many Windows
> developers are not familiar with shell or bash.
> Regarding the choice of python:
> - There are already a few instances of python usage in Hadoop, such as
> the utility (currently broken) "relnotes.py", and massive usage of
> in the examples/ and contrib/ directories.
> - Python is also used in Bigtop build-time.
> - The Python language is available for free on essentially all
> platforms, under an Apache-compatible
> - It is supported in Eclipse and similar IDEs.
> - Most importantly, it is widely accepted as a reasonably good OO
> scripting language, and it is easily learned by anyone who already knows
> shell or perl, or other common scripting languages.
> - On the Tiobe index of programming language
> which seeks to measure the relative number of software engineers who
> and use each language, Python far exceeds Perl and Ruby. The only more
> well-known scripting languages are PHP and Visual Basic, neither of
> seems a prime candidate for this use.
> For build-time usage, I think we should immediately approve python as a
> build-time dependency, and allow people who are motivated to do so, to open
> jiras for migrating existing build-time shell scripts to python.
> For run-time, there is likely to be a lot more discussion. Lots of folks,
> including me, aren't real happy with use of active scripts for
> configuration, and various others, including I believe some of the Bigtop
> folks, have issues with the way the start/stop scripts work. Nevertheless,
Radim Kolar 2012-11-21, 20:46
Konstantin Boudnik 2012-11-21, 21:33
Konstantin Boudnik 2012-11-21, 20:00
Matt Foley 2012-11-21, 21:14
Konstantin Boudnik 2012-11-21, 21:50
Andy Isaacson 2012-11-21, 23:00
Radim Kolar 2012-11-21, 23:58
Steve Loughran 2012-11-22, 09:21
Konstantin Boudnik 2012-11-22, 01:46
Radim Kolar 2012-11-22, 01:57
Chris Nauroth 2012-11-21, 21:03
Radim Kolar 2012-11-21, 21:30
Chris Nauroth 2012-11-21, 21:44
Radim Kolar 2012-11-21, 23:15
Chris Nauroth 2012-11-22, 00:14
Radim Kolar 2012-11-22, 01:55
Chris Nauroth 2012-11-22, 02:40
Radim Kolar 2012-11-22, 14:54
Steve Loughran 2012-11-22, 09:02
Matt Foley 2012-11-21, 19:44
Alejandro Abdelnur 2012-11-21, 19:58
Steve Loughran 2012-11-22, 09:14