Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> flushing + compactions after config change


+
Viral Bajaria 2013-06-27, 07:40
+
Anoop John 2013-06-27, 07:51
+
Viral Bajaria 2013-06-27, 08:18
+
谢良 2013-06-27, 08:21
+
Viral Bajaria 2013-06-27, 08:29
+
Anoop John 2013-06-27, 09:03
+
Azuryy Yu 2013-06-27, 09:22
+
Viral Bajaria 2013-06-27, 09:47
+
Azuryy Yu 2013-06-27, 09:48
+
Viral Bajaria 2013-06-27, 10:08
+
谢良 2013-06-27, 10:36
+
Azuryy Yu 2013-06-27, 14:53
+
Viral Bajaria 2013-06-27, 21:06
+
Jean-Daniel Cryans 2013-06-27, 21:40
Copy link to this message
-
Re: 答复: flushing + compactions after config change
Hey JD,

Thanks for the clarification. I also came across a previous thread which
sort of talks about a similar problem.
http://mail-archives.apache.org/mod_mbox/hbase-user/201204.mbox/%3CCAGpTDNfWNRsNqV7n3wgjE-iCHZPx-CXn1TBchgwRPOhgcoS+[EMAIL PROTECTED]%3E

I guess my problem is also similar to the fact that my writes are well
distributed and at a given time I could be writing to a lot of regions.
Some of the regions receive very little data but since the flush algorithm
choose at random what to flush when "too many hlogs" is hit, it will flush
a region with less than 10mb of data causing too many small files. This
in-turn causes compaction storms where even though major compactions is
disabled, some of the minor get upgraded to major and that's when things
start getting worse.

My compaction queues are still the same and so I doubt I will be coming out
of this storm without bumping up max hlogs for now. Reducing regions per
server is one option but then I will be wasting my resources since the
servers at current load are at < 30% CPU and < 25% RAM. Maybe I can bump up
heap space and give more memory to the the memstore. Sorry, I am just
thinking out loud.

Thanks,
Viral

On Thu, Jun 27, 2013 at 2:40 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> No, all your data eventually makes it into the log, just potentially
> not as quickly :)
>
+
Jean-Daniel Cryans 2013-06-28, 16:31
+
Viral Bajaria 2013-06-28, 21:39
+
Jean-Daniel Cryans 2013-06-28, 23:53
+
Himanshu Vashishtha 2013-07-01, 06:08
+
Azuryy Yu 2013-06-28, 01:09
+
Viral Bajaria 2013-06-28, 01:22
+
Anoop John 2013-06-28, 12:08
+
谢良 2013-06-27, 08:53
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB