Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - load a serialized object in hadoop


Copy link to this message
-
Re: load a serialized object in hadoop
Luke Lu 2010-10-13, 18:30
On Wed, Oct 13, 2010 at 8:04 AM, Shi Yu <[EMAIL PROTECTED]> wrote:
> As a coming-up to the my own question, I think to invoke the JVM in Hadoop
> requires much more memory than an ordinary JVM.

That's simply not true. The default mapreduce task Xmx is 200M, which
is much smaller than the standard jvm default 512M and most users
don't need to increase it. Please post the code reading the object (in
hdfs?) in your tasks.

> I found that instead of
> serialization the object, maybe I could create a MapFile as an index to
> permit lookups by key in Hadoop. I have also compared the performance of
> MongoDB and Memcache. I will let you know the result after I try the MapFile
> approach.
>
> Shi
>
> On 2010-10-12 21:59, M. C. Srivas wrote:
>>>
>>> On Tue, Oct 12, 2010 at 4:50 AM, Shi Yu<[EMAIL PROTECTED]>  wrote:
>>>
>>>
>>>>
>>>> Hi,
>>>>
>>>> I want to load a serialized HashMap object in hadoop. The file of stored
>>>> object is 200M. I could read that object efficiently in JAVA by setting
>>>>
>>>
>>> -Xmx
>>>
>>>>
>>>> as 1000M.  However, in hadoop I could never load it into memory. The
>>>> code
>>>>
>>>
>>> is
>>>
>>>>
>>>> very simple (just read the ObjectInputStream) and there is yet no
>>>>
>>>
>>> map/reduce
>>>
>>>>
>>>> implemented.  I set the  mapred.child.java.opts=-Xmx3000M, still get the
>>>> "java.lang.OutOfMemoryError: Java heap space"  Could anyone explain a
>>>>
>>>
>>> little
>>>
>>>>
>>>> bit how memory is allocate to JVM in hadoop. Why hadoop takes up so much
>>>> memory?  If a program requires 1G memory on a single node, how much
>>>>
>>>
>>> memory
>>>
>>>>
>>>> it requires (generally) in Hadoop?
>>>>
>>>
>>>
>>
>> The JVM reserves swap space in advance, at the time of launching the
>> process. If your swap is too low (or do not have any swap configured), you
>> will hit this.
>>
>> Or, you are on a 32-bit machine, in which case 3G is not possible in the
>> JVM.
>>
>> -Srivas.
>>
>>
>>
>>
>>>>
>>>> Thanks.
>>>>
>>>> Shi
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>