Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: DirectoryScanner's OutOfMemoryError


Copy link to this message
-
Re: DirectoryScanner's OutOfMemoryError
Please see https://issues.apache.org/jira/browse/HDFS-4461. You may
have to raise your heap for DN if you've accumulated a lot of blocks
per DN.

On Fri, Jun 7, 2013 at 8:33 PM, YouPeng Yang <[EMAIL PROTECTED]> wrote:
> Hi All
>
>    I have found that the DirectoryScanner gets error:  Error compiling
> report because of java.lang.OutOfMemoryError: Java heap space.
>   The log details are as [1]:
>
>   How does the error come out ,and how to solve this exception?
>
>
> [1]
> 2013-06-07 22:20:28,199 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Took 2737ms to process 1
> commands from NN
> 2013-06-07 22:20:28,199 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_1870928037426403148_709040 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_1870928037426403148
> 2013-06-07 22:20:28,199 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_3693882010743127822_709044 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_3693882010743127822
> 2013-06-07 22:20:28,743 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_-5452984265504491579_709036 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_-5452984265504491579
> 2013-06-07 22:20:28,743 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_-1078215880381545528_709050 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_-1078215880381545528
> 2013-06-07 22:20:28,744 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_8107220088215975918_709064 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_8107220088215975918
> 2013-06-07 22:20:29,278 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_-3527717187851336238_709052 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_-3527717187851336238
> 2013-06-07 22:20:29,812 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_1766998327682981895_709042 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_1766998327682981895
> 2013-06-07 22:20:29,812 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_1650592414141359061_709028 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_1650592414141359061
> 2013-06-07 22:20:29,812 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_-6527697040536951940_709038 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_-6527697040536951940
> 2013-06-07 22:20:43,766 ERROR
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Error compiling
> report
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)

Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB