Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> how to handle the corrupt block in HDFS?


Copy link to this message
-
Re: how to handle the corrupt block in HDFS?
more strange , in my HDFS cluster ,every block has three replicas,but i
find some one has ten replicas ,why?

# sudo -u hdfs hadoop fs -ls
/data/hisstage/helen/.staging/job_1385542328307_0915
Found 5 items
-rw-r--r--   3 helen hadoop          7 2013-11-29 14:01
/data/hisstage/helen/.staging/job_1385542328307_0915/appTokens
-rw-r--r--  10 helen hadoop    2977839 2013-11-29 14:01
/data/hisstage/helen/.staging/job_1385542328307_0915/job.jar
-rw-r--r--  10 helen hadoop       3696 2013-11-29 14:01
/data/hisstage/helen/.staging/job_1385542328307_0915/job.split
On Tue, Dec 10, 2013 at 9:15 AM, ch huang <[EMAIL PROTECTED]> wrote:

> the strange thing is when i use the following command i find 1 corrupt
> block
>
> #  curl -s http://ch11:50070/jmx |grep orrupt
>     "CorruptBlocks" : 1,
> but when i run hdfs fsck / , i get none ,everything seems fine
>
> # sudo -u hdfs hdfs fsck /
> ........
>
> ....................................Status: HEALTHY
>  Total size:    1479728140875 B (Total open files size: 1677721600 B)
>  Total dirs:    21298
>  Total files:   100636 (Files currently being written: 25)
>  Total blocks (validated):      119788 (avg. block size 12352891 B) (Total
> open file blocks (not validated): 37)
>  Minimally replicated blocks:   119788 (100.0 %)
>  Over-replicated blocks:        0 (0.0 %)
>  Under-replicated blocks:       166 (0.13857816 %)
>  Mis-replicated blocks:         0 (0.0 %)
>  Default replication factor:    3
>  Average block replication:     3.0027633
>  Corrupt blocks:                0
>  Missing replicas:              831 (0.23049656 %)
>  Number of data-nodes:          5
>  Number of racks:               1
> FSCK ended at Tue Dec 10 09:14:48 CST 2013 in 3276 milliseconds
>
> The filesystem under path '/' is HEALTHY
>
>
>  On Tue, Dec 10, 2013 at 8:32 AM, ch huang <[EMAIL PROTECTED]> wrote:
>
>> hi,maillist:
>>             my nagios alert me that there is a corrupt block in HDFS all
>> day,but i do not know how to remove it,and if the HDFS will handle this
>> automaticlly? and if remove the corrupt block will cause any data
>> lost?thanks
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB