Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> How to reduce number of entries in memory


Copy link to this message
-
Re: How to reduce number of entries in memory
Thanks for the replies. I was approaching it from a data integrity
perspective, as in wanting it flushed to disk in case of a TabletServer
failure.  Last weekend we saw two TabletServers exit the cluster due to a
network glitch, and wouldn't you know that the 04 node was secondary logger
for the 03 node.

In our case, these entries are hanging around in memory *for hours*, as the
ingest rate is not that high.

Perhaps an hourly flush of the table via the shell to get it out to disk
would be the way to go?
On Mon, Oct 28, 2013 at 4:30 PM, Mike Drob <[EMAIL PROTECTED]> wrote:

> What are you trying to accomplish by reducing the number of entries in
> memory? A tablet server will not minor compact (flush) until the native map
> fills up, but keeping things in memory isn't really a performance concern.
>
> You can force a one-time minor compaction via the shell using the 'flush'
> command.
>
>
> On Mon, Oct 28, 2013 at 5:19 PM, Terry P. <[EMAIL PROTECTED]> wrote:
>
>> Greetings all,
>> For a growing table that currently from zero to 70 million entries this
>> weekend, I'm seeing 4.4 million entries still in memory, though the client
>> programs are supposed to be flushing their entries.
>>
>> Is there a server-side setting to help reduce the number of entries that
>> are in memory (not yet flushed to disk)?  Our system has fairly light
>> performance requirements, so I'm okay if a tweak may result in reduced
>> ingest performance.
>>
>> Thanks in advance,
>> Terry
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB