Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> OutOfMemoryError: Java heap space after data load


Copy link to this message
-
OutOfMemoryError: Java heap space after data load
Greetings folks,
I have stood up our 8-node Accumulo 1.4.2 cluster consisting of 3
ZooKeepers, 1 NameNode (also runs Accumulo Master, Monitor, and GC), and 3
DataNodes / TabletServers (Secondary NameNode with Alternate Accumulo
Master process will follow).  The initial config files were copied from the
2GB/native-standalone directory.

For a quick test I have a text file I generated to load 500,000 rows of
sample data using the Accumulo shell.  For lack of a better place to run it
this first time, I ran it on the NameNode.  The script performs flushes
every 10,000 records (about 30,000 entries).  After the load finished, when
I attempt to login to the Accumulo Shell on the NameNode, I get the error:

[root@edib-namenode ~]# /usr/lib/accumulo/bin/accumulo shell -u $AUSER -p
$AUSERPWD
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 24899"...
Killed

The performance of that test was pretty poor at about 160/second (somewhat
expected, as it was just one thread) so to keep moving I generated 3
different load files and ran one on each of the 3 DataNodes /
TabletServers.  Performance was much better, sustaining 1,400 per second.
Again, the test data load files have flush commands every 10,000 records
(30,000 entries), including at the end of the file.

However, as with the NameNode, now I cannot login to the Accumulo shell on
any of the DataNodes either, as I get the same OutOfMemoryError.

My /etc/security/limits.conf file is set with 64000 for nofile and 32000
for nproc for the hdfs user (which is also running Accumulo, I haven't
split accumulo out yet).

I don't see any errors in the tserver or logger logs (standard and debug)
or any info related to the shell failing to load.  I'm at a loss with
respect to where to look.  The servers have 16GB of memory, and each has
about 14GB currently free.

Any help would be greatly appreciated.

Best regards,
Terry