Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: Child JVM memory allocation / Usage


Copy link to this message
-
Re: Child JVM memory allocation / Usage
Hi Hemanth/Koji,

Seems the above script doesn't work for me.  Can u look into the following
and suggest what more can I do
 hadoop fs -cat /user/ims-b/dump.sh
#!/bin/sh
hadoop dfs -put myheapdump.hprof /tmp/myheapdump_ims/${PWD//\//_}.hprof
hadoop jar LL.jar com.hadoop.publicationMrPOC.Launcher  Fudan\ Univ
 -Dmapred.create.symlink=yes
-Dmapred.cache.files=hdfs:///user/ims-b/dump.sh#dump.sh
-Dmapred.reduce.child.java.opts='-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./myheapdump.hprof -XX:OnOutOfMemoryError=./dump.sh'
I am not able to see the heap dump at  /tmp/myheapdump_ims

Erorr in the mapper :

Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2734)
at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
at java.util.ArrayList.add(ArrayList.java:351)
at com.hadoop.publicationMrPOC.PublicationMapper.configure(PublicationMapper.java:59)
... 22 more

On Wed, Mar 27, 2013 at 10:16 AM, Hemanth Yamijala <
[EMAIL PROTECTED]> wrote:

> Koji,
>
> Works beautifully. Thanks a lot. I learnt at least 3 different things with
> your script today !
>
> Hemanth
>
>
> On Tue, Mar 26, 2013 at 9:41 PM, Koji Noguchi <[EMAIL PROTECTED]>wrote:
>
>> Create a dump.sh on hdfs.
>>
>> $ hadoop dfs -cat /user/knoguchi/dump.sh
>> #!/bin/sh
>> hadoop dfs -put myheapdump.hprof
>> /tmp/myheapdump_knoguchi/${PWD//\//_}.hprof
>>
>> Run your job with
>>
>> -Dmapred.create.symlink=yes
>> -Dmapred.cache.files=hdfs:///user/knoguchi/dump.sh#dump.sh
>> -Dmapred.reduce.child.java.opts='-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=./myheapdump.hprof -XX:OnOutOfMemoryError=./dump.sh'
>>
>> This should create the heap dump on hdfs at /tmp/myheapdump_knoguchi.
>>
>> Koji
>>
>>
>> On Mar 26, 2013, at 11:53 AM, Hemanth Yamijala wrote:
>>
>> > Hi,
>> >
>> > I tried to use the -XX:+HeapDumpOnOutOfMemoryError. Unfortunately, like
>> I suspected, the dump goes to the current work directory of the task
>> attempt as it executes on the cluster. This directory is cleaned up once
>> the task is done. There are options to keep failed task files or task files
>> matching a pattern. However, these are NOT retaining the current working
>> directory. Hence, there is no option to get this from a cluster AFAIK.
>> >
>> > You are effectively left with the jmap option on pseudo distributed
>> cluster I think.
>> >
>> > Thanks
>> > Hemanth
>> >
>> >
>> > On Tue, Mar 26, 2013 at 11:37 AM, Hemanth Yamijala <
>> [EMAIL PROTECTED]> wrote:
>> > If your task is running out of memory, you could add the option
>> -XX:+HeapDumpOnOutOfMemoryError
>> > to mapred.child.java.opts (along with the heap memory). However, I am
>> not sure  where it stores the dump.. You might need to experiment a little
>> on it.. Will try and send out the info if I get time to try out.
>> >
>> >
>> > Thanks
>> > Hemanth
>> >
>> >
>> > On Tue, Mar 26, 2013 at 10:23 AM, nagarjuna kanamarlapudi <
>> [EMAIL PROTECTED]> wrote:
>> > Hi hemanth,
>> >
>> > This sounds interesting, will out try out that on the pseudo cluster.
>>  But the real problem for me is, the cluster is being maintained by third
>> party. I only have have a edge node through which I can submit the jobs.
>> >
>> > Is there any other way of getting the dump instead of physically going
>> to that machine and  checking out.
>> >
>> >
>> >
>> > On Tue, Mar 26, 2013 at 10:12 AM, Hemanth Yamijala <
>> [EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> > One option to find what could be taking the memory is to use jmap on
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB