These are good inputs Sanjay. Thanks for the help.
On Wed, Mar 17, 2010 at 11:33 AM, Sanjay Sharma <[EMAIL PROTECTED]
> wrote:
> Hi Ninad,
> You can always use Java object serialization to store custom objects as
> files in Hadoop distributed cache before map/reducer start running.
> The thumb rule steps of such usage is-
> a. Create the object while configuring your job, serialize it to a file and
> put it is distributed cache
> b. Initialize the same object in Map or Reducer init/setup
>
> The known issues with this approach is -
> a. additional hit of file based object
> ser-de(serialization-de-serialization). So do check whether the cumulative
> object creation time is more than the time in object ser-de
> b. works only in case of write-once read-only object
>
> In case the object size is a problem, suggest going for memcache or BDB or
> tokyotyrant or similar fast store for such distributed caching
>
> Regards,
> Sanjay Sharma
>
> -----Original Message-----
> From: Ninad Raut [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 17, 2010 10:32 AM
> To: [EMAIL PROTECTED]
> Subject: Storing Custom Java Objects in Hadoop Distibuted Cache
>
> Hi All,
>
> Currently hadoop cache only allows files to be stored in the distributed
> cache.
>
> Can I store custom objects in hadoop cache so that all my map reduce
> programs use those objects?
>
>
> Pleas let me know if any one has a solution to this problem. The Object
> creation is a very heavy task and I want to do it only once for all Map -
> Reduce Routines.
>
>
> Regards,
> Ninad .....
>
> Impetus Technologies is participating at the CTIA Wireless 2010 from 23rd
> to 25th March 2010. Meet Impetus in Las Vegas to experience our mobile and
> wireless domain expertise. Click
http://impetus.com/events to know more.
>
> Follow our updates on www.twitter.com/impetuscalling.
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
