Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # user >> Throttle major compaction.


+
Denis 2013-02-28, 13:27
+
Keith Turner 2013-02-28, 14:41
+
Denis 2013-02-28, 15:03
Copy link to this message
-
Re: Throttle major compaction.
On Thu, Feb 28, 2013 at 10:03 AM, Denis <[EMAIL PROTECTED]> wrote:
> Hi, Keith
>
> Thanks for the hint to use iterator for that.
>
> Although this approach can help in my case, it may be tricky to use it
> in other cases, because (as far as I understood), with iterator one
> can throttle the uncompressed data and actual throttling of the hard
> disk operations would depend on the data compressibility in the tablet
> being compacted and can vary widely.

Thats a good point.  I posted this on the ticket to move the discussion there.

>
> I opened the ticket ACCUMULO-1128.
>
> On 2/28/13, Keith Turner <[EMAIL PROTECTED]> wrote:
>> Can you open a ticket for this?  We could post the iterator as a patch
>> to that ticket.  Then people could use that patch.
>>
>> On Thu, Feb 28, 2013 at 8:27 AM, Denis <[EMAIL PROTECTED]> wrote:
>>> Hi.
>>>
>>> Major compaction loads hard disks very hard, even with
>>> tserver.compaction.major.concurrent.max=1
>>>
>>> Besides nice peaks on the Load Average and IOstat graps during major
>>> compaction, such high load also badly affects query performance and
>>> latency.
>>>
>>> HBase and Cassandra both have settings to throttle major compaction
>>> (Cassandra's default is 16 Mb/s which is about 1/3 of the peek values
>>> I see during Accumulo's major compaction)
>>>
>>> But it looks like Accumulo does not have such setting and I have not
>>> found relevant questions in the bugtracker and maillist archive.
>>>
>>> How do you deal with the major compaction and its performance effects?
>>
+
Keith Turner 2013-02-28, 14:26
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB