-Re: Minor compaction occurring often with fairly long delays during ingest.
Eric Newton 2013-06-26, 00:06
Increase the size of the in-memory map (24-30G), and break your table down
into multiple tablets (if you can predict your split points). This will
allow long minor compactions to start in parallel.
We have found the sweet-spot for small-record ingest to be "50-200" tablets
per server for live ingest.
You could increase the number of minor compaction threads, assuming you
have enough disks to support multiple writers.
Look for patterns: you could have an ingest hot-spot, or a slow disk/node.
On Tue, Jun 25, 2013 at 7:06 PM, Hider, Sandy <[EMAIL PROTECTED]>wrote:
> I recently setup Accumulo 1.4.2 on a rack of boxes that each has 24
> processors and 43 GB of RAM. I set them up using the 3GB example templates
> but then increased the max size of the Tserver and a few other components
> to 5GB.
> Doing some initial tests importing roughly 7000 records, each record has
> approximately 7 small fields and 1 large field holding data between 200Kb
> to 1Mb in size. While ingesting I am seeing the server hold and start
> minor compactions which seem to take quite a while after 2000-3000 records,
> and then occurring again fairly frequently
> I am wondering what options I have to try and minimize the frequency of
> minor compactions during ingest. What components memory sizes and config
> properties would help me avoid this problem? If anyone has other ideas for
> me to try and fix this please let me know.
> Thanks in advance,