Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: Object in mapreduce


Copy link to this message
-
Re: Object in mapreduce
unmesha sreeveni 2013-12-30, 04:52
AM i able to set the object in configuration?
Like conf.set("test" , myobject);

How to set object in configuration?
So while crossing mapper the value wont change right?
On Sat, Dec 28, 2013 at 6:31 PM, Chris Mawata <[EMAIL PROTECTED]>wrote:

>
> Hi Unmesha,
>      I would take a step back and look at the big picture before spending
> nights working on a non-starter. Without knowing what the ultimate goal is
> I can't give you a workaround or declare the basic architecture faulty but
> here are the things that stick out as fundamentally problematic:
>
>      1. The static keyword only gives you uniqueness in the classes loaded
> by a single ClassLoader. Once you have multiple ClassLoaders (even in the
> same JVM) use of static is almost always an error.
>
>      2. MR is a share-nothing distributed model. Apparently you intend to
> share objects and want them to stay in sync. That won't work. To do it
> would involve introducing locking and open a huge can of worms (concurency
> issues like deadlocks and race conditions) which MR spares us from.
>
>      3. What appears to be sharing (e.g. Disributed cache) is really
> sharing of class definitions and not objects. Even if you sent serialized
> objects to the Mappers they would be distinct instances.
>
>      [Apologies for only suggesting what not to do. One would have to know
> your project well to figure out what to change in the design]
>
>      Chris
>
>
> On 12/28/2013 4:59 AM, unmesha sreeveni wrote:
>
>  what i am experencing in my MR is
>
>  In Driver i am getting an object value and wen it reaches to mapper the
> object value changes.And my MR fails
> So for that is DistributedChache do this perfectly.
>
>
> On Sat, Dec 28, 2013 at 3:14 PM, Kai Voigt <[EMAIL PROTECTED]> wrote:
>
>> Check out the Distributed Cache feature:
>> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/filecache/DistributedCache.html
>>
>>  Kai
>>
>>  Am 28.12.2013 um 10:27 schrieb unmesha sreeveni <[EMAIL PROTECTED]>:
>>
>>
>>  Can we write an object from Driver class to a file and retrieve the
>> content in mapper.
>> Static is not working in mapreduce code
>>
>>  --
>> *Thanks & Regards*
>>
>>  Unmesha Sreeveni U.B
>>
>> *Junior Developer *
>> http://www.unmeshasreeveni.blogspot.in/
>>
>>
>>       ------------------------------
>> *Kai Voigt* Am Germaniahafen 1 [EMAIL PROTECTED]
>>  24143 Kiel +49 160 96683050
>>  Germany @KaiVoigt
>>
>>
>
>
>  --
> *Thanks & Regards*
>
>  Unmesha Sreeveni U.B
>
> *Junior Developer *
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
--
*Thanks & Regards*

Unmesha Sreeveni U.B

*Junior Developer*
http://www.unmeshasreeveni.blogspot.in/