Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - JNI native library loading problem in standalone mode


Copy link to this message
-
Re: JNI native library loading problem in standalone mode
Hemanth Yamijala 2010-06-01, 03:47
Edward,

If it's an option to copy the libraries to a fixed location on all the
cluster nodes, you could do that and configure them in the library
path via mapred.child.java.opts. Please look at http://bit.ly/ab93Z8
(MapReduce tutorial on Hadoop site) to see how to use this config
option for setting paths to native libraries.

Thanks
Hemanth

On Tue, Jun 1, 2010 at 6:50 AM, edward choi <[EMAIL PROTECTED]> wrote:
> Alex,
>
> Thanks for your advice.
> I tried both of them and neither works.
> I am still getting the UnsatisfiedLinkError when I try to run my application
> in StandAlone mode.
>
> I also made LangAnal.java belong to the package of "org.knu.freeparser" just
> in case. But still no luck.
> I put libetri_lmi.so to "/usr/lib" and still no luck.
>
> 2010/6/1 Alex Kozlov <[EMAIL PROTECTED]>
>
>> Try
>>
>>   1. export JAVA_LIBRARY_PATH=/home/qa/lang/dll4jni/
>>   2. mkdir lib; mv LangAnal.jar lib; jar -cvef org.etri.mp2893.LangAnal
>>   ./lib ./org
>>
>> In a distributed mode you'll have to copy your dynamic libs to some
>> directory thoughout the cluster and point JAVA_LIBRARY_PATH to it (or you
>> can use distributed cache).
>>
>> Let me know if you have other config problems.
>>
>> Alex K
>>
>> On Mon, May 31, 2010 at 9:49 AM, edward choi <[EMAIL PROTECTED]> wrote:
>>
>> > Hi, I am trying to run a JNI application on StandAlone mode and I have
>> been
>> > getting this result ever since.
>> > I've looked up every possible webs and sites but never could get the
>> > solution to it.
>> > Please help me find out what's wrong.
>> >
>> > I have 2 java files (LangAnal.java, freeparser.java)
>> > LangAnal.java belongs to a package "org.etri.mp2893"
>> > freeparser.java belongs to a package "org.knu.freeparser"
>> >
>> > freeparser.java loads "libetri_lmi.so" which is in
>> "/home/qa/lang/dll4jni/"
>> > and I believe "libetri_lmi.so" uses another library "libparser.so" which
>> is
>> > in "/home/qa/lang/lib".
>> >
>> > I have compiled LangAnal.java and freeparser.java. The resulting classes
>> > are
>> > placed respectively at "/home/qa/lang/dll4jni/org/etri/mp2893/" and
>> > "/home/qa/lang/dll4jni/org/knu/freeparser/".
>> >
>> > Then I have archived those classes by
>> >
>> > "jar -cvef org.etri.mp2893.LangAnal ./LangAnal.jar ./org/*"
>> >
>> > from the directory of "/home/qa/lang/dll4jni/".
>> >
>> > Then I tried to run
>> >
>> > hadoop jar LangAnal.jar ./news.txt ./output
>> >
>> > from the same directory and I get the result down below.
>> >
>> > I tried to set LD_LIBRARY_PATH to
>> > ".:/home/qa/lang/dll4jni:/home/qa/lang/lib"
>> > I tried java.setProperty("java.library.path", "/home/qa/lang/dll4jni");
>> in
>> > freeparser.java.
>> > So far nothing has worked.
>> >
>> > But the funny thing is, if I run
>> >
>> > hadoop -Djava.library.path=/home/qa/lang/dll4jni org.etri.mp2893.LangAnal
>> > ./news.txt ./output
>> >
>> > from "/home/qa/lang/dll4jni/" it works just fine. So I guess the problem
>> is
>> > just about jar file.
>> > And I intend to run this application in Fully-Distributed mode soon, I
>> have
>> > to figure out how to run jar files.
>> >
>> > Please someone help me.
>> >
>> >
>> >
>> -------------------------------------------------------------------------------------------------------------------
>> >
>> > [qa@qa128 dll4jni]$ hadoop jar LangAnal.jar ./news.txt ./output
>> > 10/06/01 01:19:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>> > processName=JobTracker, sessionId>> > 10/06/01 01:19:39 WARN mapred.JobClient: Use GenericOptionsParser for
>> > parsing the arguments. Applications should implement Tool for the same.
>> > 10/06/01 01:19:39 WARN mapred.JobClient: No job jar file set.  User
>> classes
>> > may not be found. See JobConf(Class) or JobConf#setJar(String).
>> > 10/06/01 01:19:39 INFO input.FileInputFormat: Total input paths to
>> process
>> > :
>> > 1
>> > 10/06/01 01:19:39 INFO mapred.JobClient: Running job: job_local_0001
>> > 10/06/01 01:19:39 INFO input.FileInputFormat: Total input paths to