Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> High cpu usage on a region server


+
OpenSource Dev 2013-09-11, 20:49
+
Stack 2013-09-11, 21:28
+
OpenSource Dev 2013-09-12, 15:15
+
lars hofhansl 2013-09-12, 15:36
+
OpenSource Dev 2013-09-12, 16:52
+
lars hofhansl 2013-09-12, 17:25
Copy link to this message
-
Re: High cpu usage on a region server
Or roll back to CDH 4.2's HBase. They are fully compatible.

J-D
On Thu, Sep 12, 2013 at 10:25 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> Not that I am aware of. Reduce the HFile block size will lessen this
> problem (but then cause other issues).
>
> It's just a fix to the RegexStringFilter. You can just recompile that and
> deploy it to the RegionServers (need to make it's in the class path before
> the HBase jars).
> Probably easier to roll a new release. It's a shame we did not see this
> earlier.
>
>
> -- Lars
>
>
>
> ________________________________
>  From: OpenSource Dev <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> Sent: Thursday, September 12, 2013 9:52 AM
> Subject: Re: High cpu usage on a region server
>
>
> Thanks Lars.
>
> Are there any other workarounds for this issue until we get the fix ?
> If not we might have to do the patch and rollout custom pkg.
>
> On Thu, Sep 12, 2013 at 8:36 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> > Yep... Very likely HBASE-9428:
> >
> > 8 threads:
> >    java.lang.Thread.State: RUNNABLE
> >         at java.util.Arrays.copyOf(Arrays.java:2786)
> >         at java.lang.StringCoding.decode(StringCoding.java:178)
> >         at java.lang.String.<init>(String.java:483)
> >         at
> org.apache.hadoop.hbase.filter.RegexStringComparator.compareTo(RegexStringComparator.java:96)
> >         ...
> >
> > 4 threads:
> >    java.lang.Thread.State: RUNNABLE
> >         at
> sun.nio.cs.ISO_8859_1$Decoder.decodeArrayLoop(ISO_8859_1.java:79)
> >         at sun.nio.cs.ISO_8859_1$Decoder.decodeLoop(ISO_8859_1.java:106)
> >         at
> java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:544)
> >         at
> java.lang.StringCoding$StringDecoder.decode(StringCoding.java:140)
> >         at java.lang.StringCoding.decode(StringCoding.java:179)
> >         at java.lang.String.<init>(String.java:483)
> >         at
> org.apache.hadoop.hbase.filter.RegexStringComparator.compareTo(RegexStringComparator.java:96)
> >
> > It's also consistent with what you see: Lots of garbage (hence tweaking
> your GC options had a significant effect)
> > The fix is in 0.94.12, which is in RC right now, probably to be released
> early next week.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: OpenSource Dev <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Sent: Thursday, September 12, 2013 8:15 AM
> > Subject: Re: High cpu usage on a region server
> >
> >
> > A server started getting busy last night, but this time it took ~5 hrs
> > to get from 15% busy to 75% busy. It is not running 80% flat-out yet.
> > But this is still very high compared to other servers that are running
> > under ~25% cpu usage. Only change that I made yesterday was the
> > addition of "-XX:+UseParNewGC" to hbase startup command.
> >
> > http://pastebin.com/VRmujgyH
> >
> > On Wed, Sep 11, 2013 at 2:28 PM, Stack <[EMAIL PROTECTED]> wrote:
> >> Can you thread dump the busy server and pastebin it?
> >> Thanks,
> >> St.Ack
> >>
> >>
> >> On Wed, Sep 11, 2013 at 1:49 PM, OpenSource Dev <
> [EMAIL PROTECTED]>wrote:
> >>
> >>> Hi,
> >>>
> >>> I'm using HBase 0.94.6 (CDH 4.3) for Opentsdb. So far I have had no
> >>> issues with writes/puts. System is handles upto 800k puts per seconds
> >>> without issue. On average we do 250k puts per second.
> >>>
> >>> I am having the problem with Reads, I've also isolated where the
> >>> problem is but not been able to find the root cause.
> >>>
> >>> I have 16 machines running hbase-region server, each has ~35 regions.
> >>> Once in a while cpu goes flatout 80% in 1 region server. These are the
> >>> things i've noticed in ganglia:
> >>>
> >>> hbase.regionserver.request - evenly distributed. Not seeing any spikes
> >>> on the busy server
> >>> hbase.regionserver.blockCacheSize - between 500MB and 1000MB
> >>> hbase.regionserver.compactionQueueSize - avg 2 or less
> >>> hbase.regionserver.blockCacheHitRatio - 30% on busy node, >60% on other
+
OpenSource Dev 2013-09-15, 06:21
+
lars hofhansl 2013-09-15, 06:30
+
OpenSource Dev 2013-09-12, 02:11
+
Ted Yu 2013-09-11, 20:57
+
OpenSource Dev 2013-09-11, 21:02
+
lars hofhansl 2013-09-11, 20:55
+
OpenSource Dev 2013-09-12, 03:59
+
lars hofhansl 2013-09-12, 04:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB