Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Hadoop native builds fail on ARM due to -m32


Copy link to this message
-
Re: Hadoop native builds fail on ARM due to -m32
Trevor Robinson 2011-05-23, 17:09
Hi Bharath,

Sorry if I was unclear: I meant that tests/benchmarks run much faster
on the Sun JRE than with OpenJDK, not necessarily faster than on other
processors. Of course, this is not surprising given that the Sun JRE
has a full JIT compiler, and OpenJDK for ARM just has the Zero C++
interpreter.

There are other, higher-performance plug-in VMs for OpenJDK (CACAO,
JamVM, Shark), but they currently have stability issues in some Linux
distributions. I believe that CACAO and JamVM are code-copying JITs,
so they place a lot of constraints on the compiler used to build them.
Shark uses the LLVM JIT, which has some serious bugs on non-x86
processors (including ARM and PPC); I believe these have not been
fixed yet because effort is focused on building a new JIT engine (the
"MC JIT") that shares more of the static compilation code:
http://blog.llvm.org/2010/04/intro-to-llvm-mc-project.html

All that said, once we have server-grade ARM hardware in the lab, I'll
certainly be looking for and sharing any performance advantages I can
find. Given the low-power, scale-out focus of these processors, it's
unlikely that we'll see higher single-thread performance than a
power-hungry x86, but we certainly to expect better performance per
watt and performance per hardware cost.

Regards,
Trevor

On Mon, May 23, 2011 at 10:46 AM, Bharath Mundlapudi
<[EMAIL PROTECTED]> wrote:
>
> Adding ARM processor support to Hadoop is great. Reducing power consumption on Hadoop grids is a plus.
>
>
> Hi Trevor,
>
> You have mentioned that - "other tests/benchmarks have run much faster". This information is good to know. Can you please tell us which areas you are seeing improvements on w.r.t ARM compared to others. Is this public information?
>
>
> -Bharath
>
>
>
> ________________________________
> From: Eli Collins <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Sunday, May 22, 2011 8:38 PM
> Subject: Re: Hadoop native builds fail on ARM due to -m32
>
> Hey Trevor,
>
> Thanks for all the info.  I took a quick look at HADOOP-7276 and
> HDFS-1920, haven't gotten a chance for a full review yet but they
> don't look like they'll be a burden, and if they get Hadoop running on
> ARM that's great!
>
> Thanks,
> Eli
>
> On Fri, May 20, 2011 at 4:27 PM, Trevor Robinson <[EMAIL PROTECTED]> wrote:
> > Hi Eli,
> >
> > On Thu, May 19, 2011 at 1:39 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
> >> Thanks for contributing.   Supporting ARM on Hadoop will require a
> >> number of different changes right? Eg given that Hadoop currently
> >> depends on some Sun-specific classes, and requires a Sun-compatible
> >> JVM you'll have to work around this dependency somehow, there's not a
> >> Sun JVM for ARM right?
> >
> > Actually, there is a Sun JVM for ARM, and it works quite well:
> >
> > http://www.oracle.com/technetwork/java/embedded/downloads/index.html
> >
> > Currently, it's just a JRE, so you have to use another JDK for javac,
> > etc., but I'm optimistic that we'll see a Sun Java SE JDK for ARM
> > servers one of these days, given all the ARM server activity from
> > Calxeda [http://www.theregister.co.uk/2011/03/14/calxeda_arm_server/],
> > Marvell, and nVidia
> > [http://www.channelregister.co.uk/2011/01/05/nvidia_arm_pc_server_chip/].
> >
> > With the patches I submitted, Hadoop builds completely and nearly all
> > of the Commons and HDFS unit tests pass with OpenJDK on ARM. (Some of
> > the Map/Reduce unit tests have some crashes due to a bug in the
> > OpenJDK build I'm using.) I need to re-run the unit tests with the Sun
> > JRE and see if they pass; other tests/benchmarks have run much faster
> > and more reliably with the Sun JRE, so I anticipate better results.
> > I've run tests like TestDFSIO with the Sun JRE and have had no
> > problems.
> >
> >> If there's a handful of additional changes then let's make an umbrella
> >> jira for Hadoop ARM support and make the issues you've already filed
> >> sub-tasks. You can ping me off-line on how to that if you want.