Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - JNI native library loading problem in standalone mode


Copy link to this message
-
JNI native library loading problem in standalone mode
edward choi 2010-05-31, 16:49
Hi, I am trying to run a JNI application on StandAlone mode and I have been
getting this result ever since.
I've looked up every possible webs and sites but never could get the
solution to it.
Please help me find out what's wrong.

I have 2 java files (LangAnal.java, freeparser.java)
LangAnal.java belongs to a package "org.etri.mp2893"
freeparser.java belongs to a package "org.knu.freeparser"

freeparser.java loads "libetri_lmi.so" which is in "/home/qa/lang/dll4jni/"
and I believe "libetri_lmi.so" uses another library "libparser.so" which is
in "/home/qa/lang/lib".

I have compiled LangAnal.java and freeparser.java. The resulting classes are
placed respectively at "/home/qa/lang/dll4jni/org/etri/mp2893/" and
"/home/qa/lang/dll4jni/org/knu/freeparser/".

Then I have archived those classes by

"jar -cvef org.etri.mp2893.LangAnal ./LangAnal.jar ./org/*"

from the directory of "/home/qa/lang/dll4jni/".

Then I tried to run

hadoop jar LangAnal.jar ./news.txt ./output

from the same directory and I get the result down below.

I tried to set LD_LIBRARY_PATH to
".:/home/qa/lang/dll4jni:/home/qa/lang/lib"
I tried java.setProperty("java.library.path", "/home/qa/lang/dll4jni"); in
freeparser.java.
So far nothing has worked.

But the funny thing is, if I run

hadoop -Djava.library.path=/home/qa/lang/dll4jni org.etri.mp2893.LangAnal
./news.txt ./output

from "/home/qa/lang/dll4jni/" it works just fine. So I guess the problem is
just about jar file.
And I intend to run this application in Fully-Distributed mode soon, I have
to figure out how to run jar files.

Please someone help me.

-------------------------------------------------------------------------------------------------------------------

[qa@qa128 dll4jni]$ hadoop jar LangAnal.jar ./news.txt ./output
10/06/01 01:19:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId10/06/01 01:19:39 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/06/01 01:19:39 WARN mapred.JobClient: No job jar file set.  User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
10/06/01 01:19:39 INFO input.FileInputFormat: Total input paths to process :
1
10/06/01 01:19:39 INFO mapred.JobClient: Running job: job_local_0001
10/06/01 01:19:39 INFO input.FileInputFormat: Total input paths to process :
1
10/06/01 01:19:39 WARN mapred.LocalJobRunner: job_local_0001
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
... 3 more
Caused by: java.lang.UnsatisfiedLinkError: no etri_lmi in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1709)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1030)
at org.knu.freeparser.freeparser.<clinit>(freeparser.java:9)
at org.etri.mp2893.LangAnal$LangAnalMapper.<init>(LangAnal.java:26)
... 8 more
10/06/01 01:19:40 INFO mapred.JobClient:  map 0% reduce 0%
10/06/01 01:19:40 INFO mapred.JobClient: Job complete: job_local_0001
10/06/01 01:19:40 INFO mapred.JobClient: Counters: 0