I am running Accumulo on some boxes with 32GB RAM (machine functions
as DataNode, TaskTracker, and TabletServer).  I have them working and
configured based on this, but I was curious if there are any rules of
thumb for configuring the memory constrained settings for Accumulo
Tablet servers.

If I wanted to devote 16GB to the various accumulo services running on
these boxes, what are the recommended RAM constrained settings for the
following configs?  (Please assume accumulo is being used for Keys
less than 100 bytes and Values less than 700 bytes).

>From accumulo-env.sh (taken from the 3GB example):

test -z "$ACCUMULO_TSERVER_OPTS" && export
ACCUMULO_TSERVER_OPTS="${POLICY} -Xmx1g -Xms1g -Xss160k"
test -z "$ACCUMULO_MASTER_OPTS"  && export
ACCUMULO_MASTER_OPTS="${POLICY} -Xmx1g -Xms1g"
test -z "$ACCUMULO_MONITOR_OPTS" && export
ACCUMULO_MONITOR_OPTS="${POLICY} -Xmx1g -Xms256m"
test -z "$ACCUMULO_GC_OPTS"      && export ACCUMULO_GC_OPTS="-Xmx256m -Xms256m"
test -z "$ACCUMULO_LOGGER_OPTS"  && export
ACCUMULO_LOGGER_OPTS="-Xmx1g -Xms256m"
test -z "$ACCUMULO_GENERAL_OPTS" && export
ACCUMULO_GENERAL_OPTS="-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75"
test -z "$ACCUMULO_OTHER_OPTS"   && export ACCUMULO_OTHER_OPTS="-Xmx1g -Xms256m"

And from accumulo-site.xml (taken from the 3GB example):

    <property>
      <name>tserver.memory.maps.max</name>
      <value>1G</value>
    </property>

    <property>
      <name>tserver.cache.data.size</name>
      <value>50M</value>
    </property>

    <property>
      <name>tserver.cache.index.size</name>
      <value>100M</value>
    </property>

Are there any others that should be changed from accumulo-site.xml?

Thanks,

--Jason

Increasing the size of the tserver and giving it larger caches will improve
query performance.

So, if you go up to 12G for your tserver, and increase your index/data
cache to 3/8G respectively, you should see improvements on warm queries.
 Keep an eye on your java gc timings, though.

Increasing the internal memory map size will improve write speed and
decrease compactions, but not by very much.

If you have a lot of tablets/files/servers, making the GC larger will keep
collection efficient.  Basically, if you can't keep the list of all files
which are candidates for deletion in memory, it has to use multiple passes
to reclaim files.

There is little benefit in making the other processes larger.

-Eric

On Sat, Aug 25, 2012 at 4:54 PM, Jason Trost <[EMAIL PROTECTED]> wrote:

> I am running Accumulo on some boxes with 32GB RAM (machine functions
> as DataNode, TaskTracker, and TabletServer).  I have them working and
> configured based on this, but I was curious if there are any rules of
> thumb for configuring the memory constrained settings for Accumulo
> Tablet servers.
>
> If I wanted to devote 16GB to the various accumulo services running on
> these boxes, what are the recommended RAM constrained settings for the
> following configs?  (Please assume accumulo is being used for Keys
> less than 100 bytes and Values less than 700 bytes).
>
> From accumulo-env.sh (taken from the 3GB example):
>
> test -z "$ACCUMULO_TSERVER_OPTS" && export
> ACCUMULO_TSERVER_OPTS="${POLICY} -Xmx1g -Xms1g -Xss160k"
> test -z "$ACCUMULO_MASTER_OPTS"  && export
> ACCUMULO_MASTER_OPTS="${POLICY} -Xmx1g -Xms1g"
> test -z "$ACCUMULO_MONITOR_OPTS" && export
> ACCUMULO_MONITOR_OPTS="${POLICY} -Xmx1g -Xms256m"
> test -z "$ACCUMULO_GC_OPTS"      && export ACCUMULO_GC_OPTS="-Xmx256m
> -Xms256m"
> test -z "$ACCUMULO_LOGGER_OPTS"  && export
> ACCUMULO_LOGGER_OPTS="-Xmx1g -Xms256m"
> test -z "$ACCUMULO_GENERAL_OPTS" && export
> ACCUMULO_GENERAL_OPTS="-XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=75"
> test -z "$ACCUMULO_OTHER_OPTS"   && export ACCUMULO_OTHER_OPTS="-Xmx1g
> -Xms256m"
>
> And from accumulo-site.xml (taken from the 3GB example):
>
>     <property>
>       <name>tserver.memory.maps.max</name>
>       <value>1G</value>
>     </property>
>
>     <property>
>       <name>tserver.cache.data.size</name>
>       <value>50M</value>
>     </property>
>
>     <property>
>       <name>tserver.cache.index.size</name>
>       <value>100M</value>
>     </property>
>
> Are there any others that should be changed from accumulo-site.xml?
>
> Thanks,
>
> --Jason
>