Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Thread safety issues with JNI/native code from map tasks?


Copy link to this message
-
Re: Thread safety issues with JNI/native code from map tasks?

On Jan 28, 2011, at 13:37 , Greg Roelofs wrote:

> Keith Wiley wrote:
>
>> However, it is also not run in any concurrent fashion except
>> w.r.t. to the JVM itself.  For example, my map task doesn't
>> make parallel calls through JNI to the native code on concurrent
>> threads at the Java level, nor does the native code itself spawn
>> any threads (like I said, it isn't even compiled with pthreads).
>
> Is speculative execution enabled?

I suppose I hadn't specifically disabled that, but two points:

(1) Speculative execution would occur on a completely different node, so there definitely isn't any thread cross-talk (in memory).  So long as they don't rely on reading/writing temp files from HDFS I don't see how they could have any effect on one another.

(2) I am also getting seg faults when I run in noncluster standalone mode, which is a single nonspeculated thread......I presume.

Can you explain your thoughts on speculative execution w.r.t. the problems I'm having?

>> So, the question is, in the scenario I have described, is there
>> any reason to suspect that the cause of my problems is some
>> sort of thread trampling between the native code and something
>> else in the surrounding environment (the JVM or something like
>> that), especially in the context of the surrounding Hadoop
>> infrastructure?  It doesn't really make any sense to me, but
>> I'm running out of ideas.
>
> I don't see any obvious possibilities except speculative execution, and
> even that would depend on how the shared library was written.  Does it
> contain any global or static variables?  If so, it's almost certainly
> not thread-safe (unless, say, a global variable were basically write-
> only and advisory only, e.g., used only in an error message or a summary
> message at the end).
Yes, not thread safe, but what difference could that make if I don't use the library in a multi-threaded fashion.  One map task, one node, one Java thread calling JNI and using the native code?  How do thread safety issues factor into this?  I admit, it's my theory that threads might be involved somehow, but I don't understand how, I'm just shooting in the dark since I can't solve this problem any other way yet.

Thanks for the input.  Can you tell me what you're thinking w.r.t. speculative execution?

I'll try it without, but I don't see how it could alter the standalone behavior.

Cheers!

________________________________________________________________________________
Keith Wiley               [EMAIL PROTECTED]               www.keithwiley.com

"Luminous beings are we, not this crude matter."
  -- Yoda
________________________________________________________________________________
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB