Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - NN Memory Jumps every 1 1/2 hours


+
Edward Capriolo 2012-12-22, 04:24
+
Adam Faris 2012-12-22, 04:59
+
Edward Capriolo 2012-12-22, 12:54
+
Michael Segel 2012-12-22, 15:42
+
Joep Rottinghuis 2012-12-22, 17:17
+
Edward Capriolo 2012-12-22, 17:51
+
Suresh Srinivas 2012-12-22, 18:32
+
Edward Capriolo 2012-12-23, 00:03
+
Edward Capriolo 2012-12-23, 01:59
+
Suresh Srinivas 2012-12-23, 03:23
+
Edward Capriolo 2012-12-23, 18:34
+
Joep Rottinghuis 2012-12-23, 19:00
+
Suresh Srinivas 2012-12-24, 02:40
Copy link to this message
-
Re: NN Memory Jumps every 1 1/2 hours
Edward Capriolo 2012-12-27, 21:48
So it turns out the issue was just the size of the filesystem.
2012-12-27 16:37:22,390 WARN
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done.
New Image Size: 4,354,340,042

Basically if the NN image size hits ~ 5,000,000,000 you get f'ed. So you
need about 3x ram as your FSImage size. If you do not have enough you die a
slow death.

On Sun, Dec 23, 2012 at 9:40 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> Do not have access to my computer. Based on reading the previous email, I
> do not see any thing suspicious on the list of objects in the histo live
> dump.
>
> I would like to hear from you about if it continued to grow. One instance
> of this I had seen in the past was related to weak reference related to
> socket objects.  I do not see that happening here though.
>
> Sent from phone
>
> On Dec 23, 2012, at 10:34 AM, Edward Capriolo <[EMAIL PROTECTED]>
> wrote:
>
> > Tried this..
> >
> > NameNode is still Ruining my Xmas on its slow death march to OOM.
> >
> > http://imagebin.org/240453
> >
> >
> > On Sat, Dec 22, 2012 at 10:23 PM, Suresh Srinivas <
> [EMAIL PROTECTED]>wrote:
> >
> >> -XX:NewSize=1G -XX:MaxNewSize=1G
>
+
Suresh Srinivas 2012-12-27, 22:08
+
Edward Capriolo 2012-12-27, 22:22
+
Suresh Srinivas 2012-12-27, 22:41
+
Edward Capriolo 2012-12-27, 22:58
+
Suresh Srinivas 2012-12-27, 23:12