Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1


Copy link to this message
-
Re: Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1
This is something you ought to raise with the IBM JVM team, as it does
appear to be a bug in their JVM.

On 21 August 2012 10:11, James F Walton <[EMAIL PROTECTED]> wrote:

> Found part of the reason while digging through UserGroupInformation.java
>
> /* Return the OS login module class name */
>   private static String getOSLoginModuleName() {
>     if (System.getProperty("java.vendor").contains("IBM")) {
>       return windows ? "com.ibm.security.auth.module.NTLoginModule"
>        : "com.ibm.security.auth.module.LinuxLoginModule";
>     } else {
>       return windows ? "com.sun.security.auth.module.NTLoginModule"
>         : "com.sun.security.auth.module.UnixLoginModule";
>     }
>   }
>
>
> So basically, if you use IBM java, then you must be on either Windows or
> Linux.  IBM's java appears to have platform specific LoginModules, there's
> AIXLoginModule for 32-bit java on AIX, and AIX64LoginModule for 64-bit java
> on AIX; however, the IBM Linux module appears not to have any 32-bit vs
> 64-bit differentiation.
>
> So, unless anyone has a means to disable this whole security setup (I'm
> not using a hadoop cluster or anything), or wants to dive headlong into
> making the necessary code changes (which I presume from my cursory scanning
> would include a little more than just the above snippet, like the
> getOSPrincipalClass as well), I guess I'll need to look at moving our
> crawler from AIX to Linux.
>
> My java coding skills are not top notch, though I could probably fix it if
> the necessary updates didn't get too convoluted beyond what I think it
> might require.  I would hope there are not too many other instances where
> AIX incompatibility would rear it's head since prior to the security
> features overhaul it all worked fine.
>
> Thoughts?
>
> James
>
>
>
>
> From:        James F Walton/Southbury/IBM@IBMUS
> To:        [EMAIL PROTECTED]
> Date:        08/17/2012 02:47 PM
> Subject:        Hadoop 1.0.3 (nutch-1.5.1) throwing errors on AIX 6.1
> ------------------------------
>
>
>
> When implementing Nutch 1.0 awhile back, we had to point our scripts to
> /opt/freeware/bin to allow Nutch's Hadoop code to utilize the more
> Linux-like versions of various system commands (like df) in order for
> Hadoop to function properly on AIX.  After doing that, we've had no issues
> with Nutch's hadoop implementation from 1.0 through 1.4.
>
> Now I'm attempting up migrate my existing, functional configurations from
> my Nutch 1.4 installation to Nutch 1.5.1, which is now using Hadoop 1.0.3.
> Now when I attempt to run a crawl, I'm getting these errors that seem to
> be coming from Hadoop, and seem to want to use a LinuxLoginModule class
> (see below)
>
> Is there some configuration setting, plugin or jar file, etc that is
> missing from the new version to make this all work on AIX again?
>
> 2012-08-02 20:03:21,271 ERROR crawl.Injector - Injector:
> java.lang.RuntimeException: java.io.IOException: failure to login
>        at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:546)
>        at
> org.apache.hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:336)
>        at org.apache.nutch.crawl.Injector.inject(Injector.java:209)
>        at org.apache.nutch.crawl.Injector.run(Injector.java:248)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.nutch.crawl.Injector.main(Injector.java:238)
> Caused by: java.io.IOException: failure to login
>        at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
>        at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452)
>        at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1494)
>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
>        at
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB