Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> NN Memory Jumps every 1 1/2 hours


+
Edward Capriolo 2012-12-22, 04:24
+
Adam Faris 2012-12-22, 04:59
+
Edward Capriolo 2012-12-22, 12:54
+
Michael Segel 2012-12-22, 15:42
+
Joep Rottinghuis 2012-12-22, 17:17
+
Edward Capriolo 2012-12-22, 17:51
+
Suresh Srinivas 2012-12-22, 18:32
+
Edward Capriolo 2012-12-23, 00:03
+
Edward Capriolo 2012-12-23, 01:59
+
Suresh Srinivas 2012-12-23, 03:23
+
Edward Capriolo 2012-12-23, 18:34
+
Joep Rottinghuis 2012-12-23, 19:00
+
Suresh Srinivas 2012-12-24, 02:40
Copy link to this message
-
Re: NN Memory Jumps every 1 1/2 hours
So it turns out the issue was just the size of the filesystem.
2012-12-27 16:37:22,390 WARN
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done.
New Image Size: 4,354,340,042

Basically if the NN image size hits ~ 5,000,000,000 you get f'ed. So you
need about 3x ram as your FSImage size. If you do not have enough you die a
slow death.

On Sun, Dec 23, 2012 at 9:40 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> Do not have access to my computer. Based on reading the previous email, I
> do not see any thing suspicious on the list of objects in the histo live
> dump.
>
> I would like to hear from you about if it continued to grow. One instance
> of this I had seen in the past was related to weak reference related to
> socket objects.  I do not see that happening here though.
>
> Sent from phone
>
> On Dec 23, 2012, at 10:34 AM, Edward Capriolo <[EMAIL PROTECTED]>
> wrote:
>
> > Tried this..
> >
> > NameNode is still Ruining my Xmas on its slow death march to OOM.
> >
> > http://imagebin.org/240453
> >
> >
> > On Sat, Dec 22, 2012 at 10:23 PM, Suresh Srinivas <
> [EMAIL PROTECTED]>wrote:
> >
> >> -XX:NewSize=1G -XX:MaxNewSize=1G
>
+
Suresh Srinivas 2012-12-27, 22:08
+
Edward Capriolo 2012-12-27, 22:22
+
Suresh Srinivas 2012-12-27, 22:41
+
Edward Capriolo 2012-12-27, 22:58
+
Suresh Srinivas 2012-12-27, 23:12
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB