Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Memory Requirements


Copy link to this message
-
Memory Requirements
Dear all,

Today I ran HBase in the pseudo-distributed mode. Since I need to save the
meta information about Web pages crawled from more than 10,000 Web sites
simultaneously and periodically. I believe the data size is large. After
running for about 30 minutes, it got the following exceptions.

     [java] ...............2012-3-10 10:09:04
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
processBatchCallback
     [java] Warning: Failed all from
region=PlayTable,,1331343367325.2bf6bf580051da504c347c19eb08f9d5.,
hostname=greatfreeinpain, port=60020
     [java] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
     [java] at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
     [java] at java.util.concurrent.FutureTask.get(FutureTask.java:83)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1557)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
     [java] at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:900)
     [java] at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:777)
     [java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:760)
     [java] at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
     [java] at
com.greatfree.hbase.AuthorityPersister.InsertInitialPlayRecord(AuthorityPersister.java:128)
     [java] at
com.greatfree.hbase.RandomPersistDispatcher.run(RandomPersistDispatcher.java:92)
     [java] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java] at java.lang.Thread.run(Thread.java:662)
     [java] Caused by: java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1371)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1383)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1381)
     [java] at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
     [java] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
     [java] ... 3 more
     [java] Caused by: java.lang.OutOfMemoryError: Java heap space
     [java] at java.util.Arrays.copyOf(Arrays.java:2786)
     [java] at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
     [java] at java.io.DataOutputStream.write(DataOutputStream.java:90)
     [java] at org.apache.hadoop.hbase.client.Put.write(Put.java:422)
     [java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
     [java] at org.apache.hadoop.hbase.client.Action.write(Action.java:94)
     [java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
     [java] at
org.apache.hadoop.hbase.client.MultiAction.write(MultiAction.java:102)
     [java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
     [java] at
org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:138)
     [java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:537)
     [java] at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:879)
     [java] at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
     [java] at $Proxy6.multi(Unknown Source)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1386)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1384)
     [java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1365)
     [java] ... 7 more

The machine running the HBase has 1G memory only over Ubuntu 10. According
to the book, HBase the Definitive Guide, 64-bit OS and 4GB are required. I
need to upgrade my system?

I change the parameter, HBASE_HEAPSIZE, from 1000 to 2048. The exceptions
still occurred. When the value was changed to 3072, it got the following
errors when I ran start-hbase.sh.

    Could not create the Java virtual machine.
    Could not create the Java virtual machine.
    starting master, logging to
/opt/hbase-0.92.0/bin/../logs/hbase-libing-master-greatfreeinpain.out
    Error occurred during initialization of VM
    Could not reserve enough space for object heap
    Could not create the Java virtual machine.
    localhost: starting regionserver, logging to
/opt/hbase-0.92.0/bin/../logs/hbase-libing-regionserver-greatfreeinpain.out
    localhost: Error occurred during initialization of VM
    localhost: Could not reserve enough space for object heap
    localhost: Could not create the Java virtual machine.

It seems that I need to increase the memory size? Do I need to install a
64-bit Ubuntu Linux?

Thanks so much for your help!

Best regards,
Bing
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB