Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Re: DirectoryScanner's OutOfMemoryError


Copy link to this message
-
Re: DirectoryScanner's OutOfMemoryError
Harsh J 2013-06-07, 16:24
Please see https://issues.apache.org/jira/browse/HDFS-4461. You may
have to raise your heap for DN if you've accumulated a lot of blocks
per DN.

On Fri, Jun 7, 2013 at 8:33 PM, YouPeng Yang <[EMAIL PROTECTED]> wrote:
> Hi All
>
>    I have found that the DirectoryScanner gets error:  Error compiling
> report because of java.lang.OutOfMemoryError: Java heap space.
>   The log details are as [1]:
>
>   How does the error come out ,and how to solve this exception?
>
>
> [1]
> 2013-06-07 22:20:28,199 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Took 2737ms to process 1
> commands from NN
> 2013-06-07 22:20:28,199 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_1870928037426403148_709040 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_1870928037426403148
> 2013-06-07 22:20:28,199 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_3693882010743127822_709044 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_3693882010743127822
> 2013-06-07 22:20:28,743 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_-5452984265504491579_709036 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_-5452984265504491579
> 2013-06-07 22:20:28,743 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_-1078215880381545528_709050 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_-1078215880381545528
> 2013-06-07 22:20:28,744 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_8107220088215975918_709064 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_8107220088215975918
> 2013-06-07 22:20:29,278 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_-3527717187851336238_709052 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_-3527717187851336238
> 2013-06-07 22:20:29,812 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_1766998327682981895_709042 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_1766998327682981895
> 2013-06-07 22:20:29,812 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_1650592414141359061_709028 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_1650592414141359061
> 2013-06-07 22:20:29,812 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted block BP-471453121-172.16.250.16-1369298226760
> blk_-6527697040536951940_709038 at file
> /home/hadoop/datadir/current/BP-471453121-172.16.250.16-1369298226760/current/finalized/subdir24/subdir25/blk_-6527697040536951940
> 2013-06-07 22:20:43,766 ERROR
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Error compiling
> report
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)

Harsh J