Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Thread safety issues with JNI/native code from map tasks?

Copy link to this message
Re: Thread safety issues with JNI/native code from map tasks?
Keith Wiley 2011-01-28, 22:52

On Jan 28, 2011, at 13:37 , Greg Roelofs wrote:

> Keith Wiley wrote:
>> However, it is also not run in any concurrent fashion except
>> w.r.t. to the JVM itself.  For example, my map task doesn't
>> make parallel calls through JNI to the native code on concurrent
>> threads at the Java level, nor does the native code itself spawn
>> any threads (like I said, it isn't even compiled with pthreads).
> Is speculative execution enabled?

I suppose I hadn't specifically disabled that, but two points:

(1) Speculative execution would occur on a completely different node, so there definitely isn't any thread cross-talk (in memory).  So long as they don't rely on reading/writing temp files from HDFS I don't see how they could have any effect on one another.

(2) I am also getting seg faults when I run in noncluster standalone mode, which is a single nonspeculated thread......I presume.

Can you explain your thoughts on speculative execution w.r.t. the problems I'm having?

>> So, the question is, in the scenario I have described, is there
>> any reason to suspect that the cause of my problems is some
>> sort of thread trampling between the native code and something
>> else in the surrounding environment (the JVM or something like
>> that), especially in the context of the surrounding Hadoop
>> infrastructure?  It doesn't really make any sense to me, but
>> I'm running out of ideas.
> I don't see any obvious possibilities except speculative execution, and
> even that would depend on how the shared library was written.  Does it
> contain any global or static variables?  If so, it's almost certainly
> not thread-safe (unless, say, a global variable were basically write-
> only and advisory only, e.g., used only in an error message or a summary
> message at the end).
Yes, not thread safe, but what difference could that make if I don't use the library in a multi-threaded fashion.  One map task, one node, one Java thread calling JNI and using the native code?  How do thread safety issues factor into this?  I admit, it's my theory that threads might be involved somehow, but I don't understand how, I'm just shooting in the dark since I can't solve this problem any other way yet.

Thanks for the input.  Can you tell me what you're thinking w.r.t. speculative execution?

I'll try it without, but I don't see how it could alter the standalone behavior.


Keith Wiley               [EMAIL PROTECTED]               www.keithwiley.com

"Luminous beings are we, not this crude matter."
  -- Yoda