Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> protobuf.ServiceException: OutOfMemoryError


Copy link to this message
-
protobuf.ServiceException: OutOfMemoryError
Hi all

  I find that some if my DNs go to dead. and the datanode log shows as [1]:

 I got the java.lang.OutOfMemoryError: Java heap space.

  I wander how this could come out.
[1]:
1496 at file
/home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_8143366700047777895
2013-06-07 13:49:35,526 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted block BP-471453121-172.16.250.16-1369298226760
blk_7448905973553274685_691506 at file
/home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_7448905973553274685
2013-06-07 13:49:35,526 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted block BP-471453121-172.16.250.16-1369298226760
blk_-1754381991139616539_691508 at file
/home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_-1754381991139616539
2013-06-07 13:49:35,527 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted block BP-471453121-172.16.250.16-1369298226760
blk_-3379745807160766937_691492 at file
/home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir56/subdir46/blk_-3379745807160766937
2013-06-07 13:51:44,231 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
java.io.IOException: com.google.protobuf.ServiceException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.ipc.ProtobufHelper.getRemoteException(ProtobufHelper.java:47)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:203)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:399)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:551)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:674)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.google.protobuf.ServiceException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)
at $Proxy10.blockReport(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy10.blockReport(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:201)
... 4 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2013-06-07 13:51:46,983 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in
block pool Block pool BP-471453121-172.16.250.16-1369298226760 (storage id
DS-1482330176-172.16.250.19-50010-1369298284371) service to wxossetl2/
172.16.250.16:8020
java.lang.OutOfMemoryError: Java heap space
2013-06-07 13:51:46,983 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service
for: Block pool BP-471453121-172.16.250.16-1369298226760 (storage id
DS-1482330176-172.16.250.19-50010-1369298284371) service to wxossetl2/
172.16.250.16:8020
2013-06-07 13:51:47,090 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool
BP-471453121-172.16.250.16-1369298226760 (storage id
DS-1482330176-172.16.250.19-50010-1369298284371)
2013-06-07 13:51:47,090 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Removed
bpid=BP-471453121-172.16.250.16-1369298226760 from blockPoolScannerMap
2013-06-07 13:51:47,090 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
Removing block pool BP-471453121-172.16.250.16-1369298226760
2013-06-07 13:51:49,091 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2013-06-07 13:51:49,098 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 0
2013-06-07 13:51:49,101 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at wxossetl5/172.16.250.19
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB