Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - executing files on hdfs via hadoop not possible? is JNI/JNA a reasonable solution?


Copy link to this message
-
executing files on hdfs via hadoop not possible? is JNI/JNA a reasonable solution?
Julian Bui 2013-03-17, 09:39
Hi hadoop users,

I just want to verify that there is no way to put a binary on HDFS and
execute it using the hadoop java api.  If not, I would appreciate advice in
getting in creating an implementation that uses native libraries.

"In contrast to the POSIX model, there are no *sticky*, *setuid* or
*setgid* bits
for files as there is no notion of executable files."  Is there no
workaround?

A little bit more about what I'm trying to do.  I have a binary that
converts my image to another image format.  I currently want to put it in
the distributed cache and tell the reducer to execute the binary on the
data on hdfs.  However, since I can't set the execute permission bit on
that file, it seems that I cannot do that.

Since I cannot use the binary, it seems like I have to use my own
implementation to do this.  The challenge is that these libraries that I
can use to do this are .a and .so files.  Would I have to use JNI and
package the libraries in the distributed cache and then have the reducer
find and use those libraries on the task nodes?  Actually, I wouldn't want
to use JNI, I'd probably want to use java native access (JNA) to do this.
 Has anyone used JNA with hadoop and been successful?  Are there problems
I'll encounter?

Please let me know.

Thanks,
-Julian