We have a 50 node cluster replicating to a 6 node cluster. Both clusters
are running CDH4.1.2 and HBase 0.94.2.

 

Today we noticed that the region servers at our replica site are using
10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh

 

These region servers have been up since January 9, 2013.

 

Does anyone have suggestions about tracking down this additional memory
use?

 

I'm not necessarily expecting the Region Server to stay right at the
12GB that we allocated, but having it running at 24GB is starting to
cause the servers to swap.

 

 
PID    USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28544  prodcon   18   0 24.1g  23g  18m S 20.0 74.0   9071:34 java
 

28544:   /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr

emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea

-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc

cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../

hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError
-Xmn256m -XX

 

 

----------------------------------------------------------------------

Ron Buckley

x6365

http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU

 

http://intranet-wiki.oclc.org/wiki/FIND

 

http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly

 

Please set "-XX:MaxDirectMemorySize" explicitly,  else the default is taking the value like -Xmx in currenty JDK6, at least for jdk1.6.30+

Best Regards,
Liang
________________________________________
发件人: Buckley,Ron [[EMAIL PROTECTED]]
发送时间: 2013年1月23日 5:17
收件人: [EMAIL PROTECTED]
主题: Region server Memory Use is double the -Xmx setting

We have a 50 node cluster replicating to a 6 node cluster. Both clusters
are running CDH4.1.2 and HBase 0.94.2.

Today we noticed that the region servers at our replica site are using
10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh

These region servers have been up since January 9, 2013.

Does anyone have suggestions about tracking down this additional memory
use?

I'm not necessarily expecting the Region Server to stay right at the
12GB that we allocated, but having it running at 24GB is starting to
cause the servers to swap.
PID    USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28544  prodcon   18   0 24.1g  23g  18m S 20.0 74.0   9071:34 java
28544:   /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr

emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea

-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc

cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../

hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError
-Xmn256m -XX

----------------------------------------------------------------------

Ron Buckley

x6365

http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU

http://intranet-wiki.oclc.org/wiki/FIND

http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly

Liang,

Thanks.  I wasn’t really aware that the direct memory could get that large. (Full disclosure, we did switch from jdk1.6.0_25 to jdk1.6.0_31 the last time we restarted HBase.)

I've only seen explicit setting of -XX:MaxDirectMemorySize for regionservers associated with the experimental off-heap cache.

Is anyone else running their region servers with -XX:MaxDirectMemorySize (not using the off-heap cache)?

Ron

-----Original Message-----
From: 谢良 [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, January 22, 2013 9:20 PM
To: [EMAIL PROTECTED]
Subject: 答复: Region server Memory Use is double the -Xmx setting

Please set "-XX:MaxDirectMemorySize" explicitly,  else the default is taking the value like -Xmx in currenty JDK6, at least for jdk1.6.30+

Best Regards,
Liang
________________________________________
发件人: Buckley,Ron [[EMAIL PROTECTED]]
发送时间: 2013年1月23日 5:17
收件人: [EMAIL PROTECTED]
主题: Region server Memory Use is double the -Xmx setting

We have a 50 node cluster replicating to a 6 node cluster. Both clusters
are running CDH4.1.2 and HBase 0.94.2.

Today we noticed that the region servers at our replica site are using
10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh

These region servers have been up since January 9, 2013.

Does anyone have suggestions about tracking down this additional memory
use?

I'm not necessarily expecting the Region Server to stay right at the
12GB that we allocated, but having it running at 24GB is starting to
cause the servers to swap.
PID    USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28544  prodcon   18   0 24.1g  23g  18m S 20.0 74.0   9071:34 java
28544:   /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr

emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea

-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc

cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../

hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError
-Xmn256m -XX

----------------------------------------------------------------------

Ron Buckley

x6365

http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU

http://intranet-wiki.oclc.org/wiki/FIND

http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly

Are  you using compression for HFiles?

Yes we are using  "MaxDirectMemorySize" and we dont use off-heap cache.

-Anoop-
________________________________________
From: Buckley,Ron [[EMAIL PROTECTED]]
Sent: Wednesday, January 23, 2013 8:49 PM
To: [EMAIL PROTECTED]
Subject: RE: Region server Memory Use is double the -Xmx setting

Liang,

Thanks.  I wasn’t really aware that the direct memory could get that large. (Full disclosure, we did switch from jdk1.6.0_25 to jdk1.6.0_31 the last time we restarted HBase.)

I've only seen explicit setting of -XX:MaxDirectMemorySize for regionservers associated with the experimental off-heap cache.

Is anyone else running their region servers with -XX:MaxDirectMemorySize (not using the off-heap cache)?

Ron

-----Original Message-----
From: 谢良 [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, January 22, 2013 9:20 PM
To: [EMAIL PROTECTED]
Subject: 答复: Region server Memory Use is double the -Xmx setting

Please set "-XX:MaxDirectMemorySize" explicitly,  else the default is taking the value like -Xmx in currenty JDK6, at least for jdk1.6.30+

Best Regards,
Liang
________________________________________
发件人: Buckley,Ron [[EMAIL PROTECTED]]
发送时间: 2013年1月23日 5:17
收件人: [EMAIL PROTECTED]
主题: Region server Memory Use is double the -Xmx setting

We have a 50 node cluster replicating to a 6 node cluster. Both clusters
are running CDH4.1.2 and HBase 0.94.2.

Today we noticed that the region servers at our replica site are using
10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh

These region servers have been up since January 9, 2013.

Does anyone have suggestions about tracking down this additional memory
use?

I'm not necessarily expecting the Region Server to stay right at the
12GB that we allocated, but having it running at 24GB is starting to
cause the servers to swap.
PID    USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28544  prodcon   18   0 24.1g  23g  18m S 20.0 74.0   9071:34 java
28544:   /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr

emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea

-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc

cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../

hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError
-Xmn256m -XX

----------------------------------------------------------------------

Ron Buckley

x6365

http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU

http://intranet-wiki.oclc.org/wiki/FIND

http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly

Anoop,

We use Snappy compression for all our tables.

I tried '-XX:MaxDirectMemorySize=1g' on our test cluster last night.

Ran great until major compactions started, then all region servers took 'java.lang.OutOfMemoryError: Direct buffer memory'

Trying '-XX:MaxDirectMemorySize=2g' tonight.
Ron

-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, January 23, 2013 10:22 PM
To: [EMAIL PROTECTED]
Subject: RE: Region server Memory Use is double the -Xmx setting

Are  you using compression for HFiles?

Yes we are using  "MaxDirectMemorySize" and we dont use off-heap cache.

-Anoop-
________________________________________
From: Buckley,Ron [[EMAIL PROTECTED]]
Sent: Wednesday, January 23, 2013 8:49 PM
To: [EMAIL PROTECTED]
Subject: RE: Region server Memory Use is double the -Xmx setting

Liang,

Thanks.  I wasn’t really aware that the direct memory could get that large. (Full disclosure, we did switch from jdk1.6.0_25 to jdk1.6.0_31 the last time we restarted HBase.)

I've only seen explicit setting of -XX:MaxDirectMemorySize for regionservers associated with the experimental off-heap cache.

Is anyone else running their region servers with -XX:MaxDirectMemorySize (not using the off-heap cache)?

Ron

-----Original Message-----
From: 谢良 [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, January 22, 2013 9:20 PM
To: [EMAIL PROTECTED]
Subject: 答复: Region server Memory Use is double the -Xmx setting

Please set "-XX:MaxDirectMemorySize" explicitly,  else the default is taking the value like -Xmx in currenty JDK6, at least for jdk1.6.30+

Best Regards,
Liang
________________________________________
发件人: Buckley,Ron [[EMAIL PROTECTED]]
发送时间: 2013年1月23日 5:17
收件人: [EMAIL PROTECTED]
主题: Region server Memory Use is double the -Xmx setting

We have a 50 node cluster replicating to a 6 node cluster. Both clusters are running CDH4.1.2 and HBase 0.94.2.

Today we noticed that the region servers at our replica site are using 10GB more memory than the '-Xmx12288m' we have defined in hbase-env.sh

These region servers have been up since January 9, 2013.

Does anyone have suggestions about tracking down this additional memory use?

I'm not necessarily expecting the Region Server to stay right at the 12GB that we allocated, but having it running at 24GB is starting to cause the servers to swap.
PID    USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28544  prodcon   18   0 24.1g  23g  18m S 20.0 74.0   9071:34 java
28544:   /usr/java/jdk1.6.0_31/bin/java -XX:OnOutOfMemoryError=kill -9
%p -Xmx12288m -Dcom.sun.management.jmxr

emote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=9021 -ea

-server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOc

cupancyFraction=70 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/drive1/hadoop/2.0/isoft/../

hbase/logs/gc-hbase.log -ea -server -XX:+HeapDumpOnOutOfMemoryError -Xmn256m -XX

----------------------------------------------------------------------

Ron Buckley

x6365

http://intranet-wiki.oclc.org/wiki/XWC/XServe/SRU

http://intranet-wiki.oclc.org/wiki/FIND

http://intranet-wiki.oclc.org/wiki/Higgins
http://intranet-wiki.oclc.org/wiki/Firefly