Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Is hadoop tread safe?


Copy link to this message
-
Re: Is hadoop tread safe?
Harsh J 2013-08-29, 15:19
Map tasks run in parallel spawned JVMs, so are isolated from one
another at runtime. Use of static functions shouldn't affect you
generally.

Default Map I/O is single-threaded. If you plan to use
multiple-threads, use MultiThreadedMapper for proper thread-safety.

On Thu, Aug 29, 2013 at 8:15 PM, Pavan Sudheendra <[EMAIL PROTECTED]> wrote:
> No, I had written a huge Map Reduce program which talks with hbase and does
> a lot of computing using it as a source as well as sink.. One of my
> colleague saw my code and saw that I had used a lot of static function
> instead of making use of proper OOP concepts.. He was telling me that it
> shouldn't be the way I should go about doing it.. But my code works fine..
> So, was wondering will I face any problem in the future because of this..
> That's all.
>
> Regards,
> Pavan
>
> On Aug 29, 2013 8:11 PM, "Adam Muise" <[EMAIL PROTECTED]> wrote:
>>
>> Mappers don't communicate with each other in traditional MapReduce. If you
>> need something more MPI-ish then look to MPI over YARN or write your own
>> YARN app.
>>
>> If you need multi-threading within the mapper then it is up to you as the
>> java developer to make it thread safe. Use the concurrent libraries like
>> anything else and Bob's your uncle. Having overly-complicated mappers can be
>> difficult to manage however and it kind of misses the mark for MapReduce
>> problems.
>>
>> Maybe if you expand on your use case a bit someone here can provide
>> specific advice.
>>
>>
>> On Thu, Aug 29, 2013 at 10:33 AM, Pavan Sudheendra <[EMAIL PROTECTED]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> Is hadoop thread safe? Do mappers make use of threads in any chance? A
>>> little bit of information on the way they execute in parallel would help me
>>> out. Thanks.
>>>
>>> Regards,
>>> Pavan
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> [EMAIL PROTECTED]
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.

--
Harsh J