Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Object in mapreduce


+
unmesha sreeveni 2013-12-28, 09:27
+
Kai Voigt 2013-12-28, 09:44
+
unmesha sreeveni 2013-12-28, 09:59
Copy link to this message
-
Re: Object in mapreduce
Chris Mawata 2013-12-28, 13:01

Hi Unmesha,
      I would take a step back and look at the big picture before
spending nights working on a non-starter. Without knowing what the
ultimate goal is I can't give you a workaround or declare the basic
architecture faulty but here are the things that stick out as
fundamentally problematic:

      1. The static keyword only gives you uniqueness in the classes
loaded by a single ClassLoader. Once you have multiple ClassLoaders
(even in the same JVM) use of static is almost always an error.

      2. MR is a share-nothing distributed model. Apparently you intend
to share objects and want them to stay in sync. That won't work. To do
it would involve introducing locking and open a huge can of worms
(concurency issues like deadlocks and race conditions) which MR spares
us from.

      3. What appears to be sharing (e.g. Disributed cache) is really
sharing of class definitions and not objects. Even if you sent
serialized objects to the Mappers they would be distinct instances.

      [Apologies for only suggesting what not to do. One would have to
know your project well to figure out what to change in the design]

      Chris

On 12/28/2013 4:59 AM, unmesha sreeveni wrote:
> what i am experencing in my MR is
>
> In Driver i am getting an object value and wen it reaches to mapper
> the object value changes.And my MR fails
> So for that is DistributedChache do this perfectly.
>
>
> On Sat, Dec 28, 2013 at 3:14 PM, Kai Voigt <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Check out the Distributed Cache feature:
>     http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/filecache/DistributedCache.html
>
>
>     Kai
>
>     Am 28.12.2013 um 10:27 schrieb unmesha sreeveni
>     <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>:
>
>>
>>     Can we write an object from Driver class to a file and retrieve
>>     the content in mapper.
>>     Static is not working in mapreduce code
>>
>>     --
>>     /Thanks & Regards/
>>     /
>>     /
>>     Unmesha Sreeveni U.B/
>>     /
>>     /Junior Developer
>>     /
>>     http://www.unmeshasreeveni.blogspot.in/
>>     /
>>     /
>
>     ------------------------------------------------------------------------
>     *Kai Voigt*Am Germaniahafen [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>     24143 Kiel+49 160 96683050
>     Germany@KaiVoigt
>
>
>
>
> --
> /Thanks & Regards/
> /
> /
> Unmesha Sreeveni U.B/
> /
> /Junior Developer
> /
> http://www.unmeshasreeveni.blogspot.in/
> /
> /