Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> MPI: Java/JNI help


Copy link to this message
-
Re: MPI: Java/JNI help
I was able to dig further into this, and we believe we finally tracked this down to root cause. It appears that java loads things into a private, as opposed to global, namespace. Thus, the Java MPI bindings load the initial libmpi just fine.

However, when libmpi then attempts to load the individual plug-ins beneath it, the load fails due to "unfound" symbols. Our plug-ins are implemented as individual dll's, and reference symbols from within the larger libmpi above them. In order to find those symbols, the libraries must be in the global namespace.

We have a workaround - namely, to disable dlopen so all the plug-ins get pulled up into libmpi. However, this eliminates the ability for a vendor to distribute a binary, proprietary plug-in that we "absorb" during dlopen. For the moment, this isn't a big deal, but it could be an issue down the line. We have some ideas on how to resolve it internally, but it would take a fair amount of work, and have some side-effects.

Does anyone know if it is possible to convince java to use the global namespace? Or can you point me to someone/someplace where I should explore the question?

Thanks
Ralph

On Jan 30, 2012, at 5:13 PM, Kihwal Lee wrote:

> It doesn't have to be static.
> Do architectures match between the node manager jvm and the library?
> If one is 32 bit and the other is 64, it won't work.
>
> Kihwal
>
> On 1/30/12 5:58 PM, "Ralph Castain" <[EMAIL PROTECTED]> wrote:
>
> Hi folks
>
> As per earlier emails, I'm just about ready to release the Java MPI bindings. I have one remaining issue and would appreciate some help.
>
> We typically build OpenMPI dynamically. For the Java bindings, this means that the JNI code underlying the Java binding must dynamically load OMPI plug-ins. Everything works fine on Mac. However, on Linux, I am getting dynamic library load errors.
>
> I have tried setting -Djava.library.path and LD_LIBRARY_PATH to the correct locations. In both cases, I get errors from the JNI code indicating that it was unable to open the specified dynamic library.
>
> I have heard from one person that JNI may need to be built statically, and I suppose it is possible that Apple's customized Java implementation specifically resolved that problem. However, all the online documentation I can find indicates that Java on Linux should also be able to load dynamic libraries - but JNI is not specifically addressed.
>
> Can any of you Java experts provide advice on this behavior? I'd like to get these bindings released!
>
> Thanks
> Ralph
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB