Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> mslab enabled jvm crash


Copy link to this message
-
Re: mslab enabled jvm crash
Wayne, I think you are hitting fragmentation, how often do you flush?
 Can you share memstore flush graphs?
Here is ours:  http://img851.yfrog.com/img851/9814/screenshot20110526at124.png

We run at 12G Heap, 20% memstore size, 50% blockcache, have recently
added incremental mode to combat too frequent ParNew GCs:

export HBASE_OPTS="$HBASE_OPTS -verbose:gc -Xms12000m
-XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
-Xloggc:$HBASE_HOME/logs/gc-hbase.log \
-XX:+CMSIncrementalMode \
-XX:+CMSIncrementalPacing \
-XX:-TraceClassUnloading
"

-Jack

On Wed, May 25, 2011 at 5:55 PM, Wayne <[EMAIL PROTECTED]> wrote:
> We are using std thrift from python. All writes are batched into usually 30k
> writes per batch. The writes are small double/varchar(100) type values. Our
> current write performance is fine for our needs...our concern is that they
> are not sustainable over time given the GC timeouts.
>
> Per the 4 items above, yes the staff is biased (obviously), the OS is what
> 2/3+ linux servers are running, the machines are Super Micro twins with 6
> sata 1TB RE4 disks (5 for hdfs) and 24G of ECC memory (hardly non-standard).
> We are plain vanilla. Our workload may be non-standard in that it is NOT
> map-reduce. It is an evenly distributed q based home spun parallel
> processing framework.
>
> We are not a Java shop, and do not want to become one. I think to push the
> limits and do well with hadoop+hdfs you have to buy into Java and have deep
> skills there. We are database experts looking only for an open source
> scalable database. We are a python shop and have no interest in digging deep
> into java and the jvm.
>
> Thanks.
>
> On Wed, May 25, 2011 at 8:07 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>
>> How large are these writes?
>>
>> Are you using asynchbase or other alternative client implementation?
>>
>> Are you batching updates?
>>
>> On Wed, May 25, 2011 at 2:44 PM, Wayne <[EMAIL PROTECTED]> wrote:
>>
>> > What are your write levels? We are pushing 30-40k writes/sec/node on 10
>> > nodes for 24-36-48-72 hours straight. We have only 4 writers per node so
>> we
>> > are hardly overwhelming the nodes. Disk utilization runs at 10-20%, load
>> is
>> > max 50% including some app code, and memory is the 8g JVM out of 24G. We
>> > run
>> > our production as 20TB in MySQL and see 90% disk utilization for hours
>> > every
>> > day. A database must be able to accept being pounded 24/7. If the
>> hardware
>> > can handle it so should the database...this is not true for Java based
>> > databases. The reality is that Java is not nearly ready for what a real
>> > database will expect of it. Sure we could "back off" the volume and add
>> 100
>> > more nodes like Cassandra requires but then we might as well have used
>> > something else given that hardware spend.
>> >
>> > Our problem is that we have invested so much time with Hbase that it is
>> > hard
>> > for us to walk away and go to the sharded PostgreSQL we should have used
>> 9
>> > months back. Sorry for the negativity but considering giving up after
>> > having
>> > invested all of this time is painful.
>> >
>> >
>> > On Wed, May 25, 2011 at 4:21 PM, Erik Onnen <[EMAIL PROTECTED]> wrote:
>> >
>> > > On Wed, May 25, 2011 at 11:39 AM, Ted Dunning <[EMAIL PROTECTED]>
>> > > wrote:
>> > > > It should be recognized that your experiences are a bit out of the
>> norm
>> > > > here.  Many hbase installations use more recent JVM's without
>> problems.
>> > >
>> > > Indeed, we run u25 on CentOS 5.6 and over several days uptime it's
>> > > common to never see a full GC across an 8GB heap.
>> > >
>> > > What we never see is a ParNew taking .1 seconds, they're usually .01
>> > > and we never have full collections lasting 92 seconds. The only time
>> > > I've ever seen a JVM at 8GB take that long is when running on puny
>> > > (read virtualized) cores or when there are Ubuntu kernel bugs at play.
>> > > The same is true for our Cassandra deploys as well.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB