Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1


Copy link to this message
-
Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1
This is something you ought to raise with the IBM JVM team, as it does
appear to be a bug in their JVM.

On 21 August 2012 10:11, James F Walton <[EMAIL PROTECTED]> wrote:

> Found part of the reason while digging through UserGroupInformation.java
>
> /* Return the OS login module class name */
>   private static String getOSLoginModuleName() {
>     if (System.getProperty("java.vendor").contains("IBM")) {
>       return windows ? "com.ibm.security.auth.module.NTLoginModule"
>        : "com.ibm.security.auth.module.LinuxLoginModule";
>     } else {
>       return windows ? "com.sun.security.auth.module.NTLoginModule"
>         : "com.sun.security.auth.module.UnixLoginModule";
>     }
>   }
>
>
> So basically, if you use IBM java, then you must be on either Windows or
> Linux.  IBM's java appears to have platform specific LoginModules, there's
> AIXLoginModule for 32-bit java on AIX, and AIX64LoginModule for 64-bit java
> on AIX; however, the IBM Linux module appears not to have any 32-bit vs
> 64-bit differentiation.
>
> So, unless anyone has a means to disable this whole security setup (I'm
> not using a hadoop cluster or anything), or wants to dive headlong into
> making the necessary code changes (which I presume from my cursory scanning
> would include a little more than just the above snippet, like the
> getOSPrincipalClass as well), I guess I'll need to look at moving our
> crawler from AIX to Linux.
>
> My java coding skills are not top notch, though I could probably fix it if
> the necessary updates didn't get too convoluted beyond what I think it
> might require.  I would hope there are not too many other instances where
> AIX incompatibility would rear it's head since prior to the security
> features overhaul it all worked fine.
>
> Thoughts?
>
> James
>
>
>
>
> From:        James F Walton/Southbury/IBM@IBMUS
> To:        [EMAIL PROTECTED]
> Date:        08/17/2012 02:47 PM
> Subject:        Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1
> ------------------------------
>
>
>
> When implementing Nutch 1.0 awhile back, we had to point our scripts to
> /opt/freeware/bin to allow Nutch's Hadoop code to utilize the more
> Linux-like versions of various system commands (like df) in order for
> Hadoop to function properly on AIX.  After doing that, we've had no issues
> with Nutch's hadoop implementation from 1.0 through 1.4.
>
> Now I'm attempting up migrate my existing, functional configurations from
> my Nutch 1.4 installation to Nutch 1.5.1, which is now using Hadoop 1.0.3.
> Now when I attempt to run a crawl, I'm getting these errors that seem to
> be coming from Hadoop, and seem to want to use a LinuxLoginModule class
> (see below)
>
> Is there some configuration setting, plugin or jar file, etc that is
> missing from the new version to make this all work on AIX again?
>
> 2012-08-02 20:03:21,271 ERROR crawl.Injector - Injector:
> java.lang.RuntimeException: java.io.IOException: failure to login
>        at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:546)
>        at
> org.apache.hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:336)
>        at org.apache.nutch.crawl.Injector.inject(Injector.java:209)
>        at org.apache.nutch.crawl.Injector.run(Injector.java:248)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.nutch.crawl.Injector.main(Injector.java:238)
> Caused by: java.io.IOException: failure to login
>        at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
>        at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452)
>        at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1494)
>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
>        at