-Re: Thread safety issues with JNI/native code from map tasks?
Keith Wiley 2011-01-28, 22:52
On Jan 28, 2011, at 13:37 , Greg Roelofs wrote:
> Keith Wiley wrote:
>> However, it is also not run in any concurrent fashion except
>> w.r.t. to the JVM itself. For example, my map task doesn't
>> make parallel calls through JNI to the native code on concurrent
>> threads at the Java level, nor does the native code itself spawn
>> any threads (like I said, it isn't even compiled with pthreads).
> Is speculative execution enabled?
I suppose I hadn't specifically disabled that, but two points:
(1) Speculative execution would occur on a completely different node, so there definitely isn't any thread cross-talk (in memory). So long as they don't rely on reading/writing temp files from HDFS I don't see how they could have any effect on one another.
(2) I am also getting seg faults when I run in noncluster standalone mode, which is a single nonspeculated thread......I presume.
Can you explain your thoughts on speculative execution w.r.t. the problems I'm having?
>> So, the question is, in the scenario I have described, is there
>> any reason to suspect that the cause of my problems is some
>> sort of thread trampling between the native code and something
>> else in the surrounding environment (the JVM or something like
>> that), especially in the context of the surrounding Hadoop
>> infrastructure? It doesn't really make any sense to me, but
>> I'm running out of ideas.
> I don't see any obvious possibilities except speculative execution, and
> even that would depend on how the shared library was written. Does it
> contain any global or static variables? If so, it's almost certainly
> not thread-safe (unless, say, a global variable were basically write-
> only and advisory only, e.g., used only in an error message or a summary
> message at the end).
Yes, not thread safe, but what difference could that make if I don't use the library in a multi-threaded fashion. One map task, one node, one Java thread calling JNI and using the native code? How do thread safety issues factor into this? I admit, it's my theory that threads might be involved somehow, but I don't understand how, I'm just shooting in the dark since I can't solve this problem any other way yet.
Thanks for the input. Can you tell me what you're thinking w.r.t. speculative execution?
I'll try it without, but I don't see how it could alter the standalone behavior.
Keith Wiley [EMAIL PROTECTED] www.keithwiley.com
"Luminous beings are we, not this crude matter."