Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # general - [VOTE] Release Apache Hadoop 0.23.1-rc1
Arun C Murthy
2012-02-15, 05:28
Robert Evans
2012-02-16, 21:51
Vinod Kumar Vavilapalli
2012-02-16, 23:04
Robert Evans
2012-02-17, 16:20
Thomas Graves
2012-02-17, 19:14
Arun C Murthy
2012-02-18, 01:31
Arun C Murthy
2012-02-18, 07:23
Ted Yu
2012-02-18, 21:16
Taeseong Kim
2012-02-19, 04:22
Thomas Graves
2012-02-22, 01:11
Mahadev Konar
2012-02-22, 07:45
Robert Evans
2012-02-22, 19:30
Roman Shaposhnik
2012-02-23, 01:42
Todd Lipcon
2012-02-23, 02:43
Vinod Kumar Vavilapalli
2012-02-23, 03:45
Vinod Kumar Vavilapalli
2012-02-23, 03:51
Eli Collins
2012-02-23, 04:30
Eli Collins
2012-02-23, 04:48
Todd Lipcon
2012-02-23, 05:00
Arun C Murthy
2012-02-23, 05:32
Mahadev Konar
2012-02-23, 05:37
Suresh Srinivas
2012-02-23, 05:38
Tom White
2012-02-24, 19:04
Vinod Kumar Vavilapalli
2012-02-24, 19:36
Siddharth Seth
2012-02-25, 02:04
Thomas Graves
2013-04-16, 13:43
Copy link to this message
-
Re: [VOTE] Release Apache Hadoop 0.23.1-rc2
Roman Shaposhnik 2012-02-23, 04:48
On Wed, Feb 22, 2012 at 7:45 PM, Vinod Kumar Vavilapalli
<[EMAIL PROTECTED]> wrote:
> Roman,
>
> you are compiling on 64 bit machines?

I'm compiling on both 32bit and 64bit architectures.

>> not only make it impossible to build the right version
>> of container executor without hand-editing the pom.xml,
>
> The idea was to have 32 bit binary all the time

This is a flawed idea and that's one half of the reason I gave this RC
a -1. You can NOT force all 64bit environments to always have
a 32bit set of libraries, etc. Basically, with this release, all of a sudden
you force all the deployments to install quite a lot of extra RPMs/DEBs
for no other reason but to workaround a deficiency in the default Hadoop
build. Now, that would been enough to give this RC a -0 but
an added issue of prohibiting regeneration of the correct packages with the
reasonable set of binaries makes it -1.

> can you let us know what issue you are facing. During the build itself?

http://bigtop01.cloudera.org:8080/view/Bigtop-hadoop-0.23/job/Bigtop-hadoop-0.23-Hadoop/12/

Click on any of the distributions to see how the build fails.

Note -- these are clean VMs. We take special precautions
not to pollute them with any extra dependencies. These VMs are
what distribution providers (CentOS, SLES, Canonical) have deemed
to be the worthy of public consumption on Amazon's EC2.

>> but also make it impossible to deploy the secure
>> version of hadoop without additional burden of installing
>> 32bit legacy support on all the slave nodes.
>
> Again, what do you mean by this? You mean you will have to install all
> 32 bit dependencies that the container-executor may have on all nodes?

That is correct. These dependencies are not available by default, have
an extra layer of complexity and are being slowly phased out of support
(to the best of my knowledge).

> AFAIU, if you have a single 32bit lib/binary on a 64 bit machine, you
> don't need any other 32bit lib dependencies. Please correct me if I am
> wrong.

$ tar xzvf hadoop-0.23.1.tag.gz
$ ldd bin/container-executor
linux-gate.so.1 =>  (0xf770a000)
libc.so.6 => /lib32/libc.so.6 (0xf758d000)
/lib/ld-linux.so.2 (0xf770b000)

That /lib32/libc.so.6 dependency is a problem.

> Myself cannot test it exactly like the way you are testing as I
> don't have access to nodes which don't have 32bits installed, so more
> specific details are helpful.

You don't have to go far to get a testbed -- any default 64bit VM
of all the major Linux distributions on EC2 would suffice.

Thanks,
Roman.