Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> MPI: Java/JNI help


+
Ralph Castain 2012-01-30, 23:58
+
Kihwal Lee 2012-01-31, 00:13
+
Ralph Castain 2012-01-31, 00:37
+
Ralph Castain 2012-01-31, 19:07
+
Ralph Castain 2012-01-31, 22:34
+
Kihwal Lee 2012-01-31, 23:35
Copy link to this message
-
Re: MPI: Java/JNI help
Thanks Kihwal! This was an excellent suggestion and worked great!

Should have the Java bindings out shortly...

On Jan 31, 2012, at 4:35 PM, Kihwal Lee wrote:

> There might be other tricks you can play with CL, but here is my idea: You could have the initial jni native lib to become a sort of wrapper to dlopen() the real thing (the one plug-ins depend on) with RTLD_GLOBAL, so that the fact that the jni library is loaded in a specific name space does not matter.
>
> Kihwal
>
> On 1/31/12 4:34 PM, "Ralph Castain" <[EMAIL PROTECTED]> wrote:
>
> I was able to dig further into this, and we believe we finally tracked this down to root cause. It appears that java loads things into a private, as opposed to global, namespace. Thus, the Java MPI bindings load the initial libmpi just fine.
>
> However, when libmpi then attempts to load the individual plug-ins beneath it, the load fails due to "unfound" symbols. Our plug-ins are implemented as individual dll's, and reference symbols from within the larger libmpi above them. In order to find those symbols, the libraries must be in the global namespace.
>
> We have a workaround - namely, to disable dlopen so all the plug-ins get pulled up into libmpi. However, this eliminates the ability for a vendor to distribute a binary, proprietary plug-in that we "absorb" during dlopen. For the moment, this isn't a big deal, but it could be an issue down the line. We have some ideas on how to resolve it internally, but it would take a fair amount of work, and have some side-effects.
>
> Does anyone know if it is possible to convince java to use the global namespace? Or can you point me to someone/someplace where I should explore the question?
>
> Thanks
> Ralph
>
> On Jan 30, 2012, at 5:13 PM, Kihwal Lee wrote:
>
>> It doesn't have to be static.
>> Do architectures match between the node manager jvm and the library?
>> If one is 32 bit and the other is 64, it won't work.
>>
>> Kihwal
>>
>> On 1/30/12 5:58 PM, "Ralph Castain" <[EMAIL PROTECTED]> wrote:
>>
>> Hi folks
>>
>> As per earlier emails, I'm just about ready to release the Java MPI bindings. I have one remaining issue and would appreciate some help.
>>
>> We typically build OpenMPI dynamically. For the Java bindings, this means that the JNI code underlying the Java binding must dynamically load OMPI plug-ins. Everything works fine on Mac. However, on Linux, I am getting dynamic library load errors.
>>
>> I have tried setting -Djava.library.path and LD_LIBRARY_PATH to the correct locations. In both cases, I get errors from the JNI code indicating that it was unable to open the specified dynamic library.
>>
>> I have heard from one person that JNI may need to be built statically, and I suppose it is possible that Apple's customized Java implementation specifically resolved that problem. However, all the online documentation I can find indicates that Java on Linux should also be able to load dynamic libraries - but JNI is not specifically addressed.
>>
>> Can any of you Java experts provide advice on this behavior? I'd like to get these bindings released!
>>
>> Thanks
>> Ralph
>>
>>
>
>
+
Noah Watkins 2012-01-31, 00:24
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB