Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> log


+
Mohit Vadhera 2013-04-19, 06:38
+
S, Manoj 2013-04-19, 06:58
+
Nitin Pawar 2013-04-19, 06:44
+
Bejoy Ks 2013-04-19, 09:21
But my files are still there. As an example

-rw-r--r--  10 User hadmin    6392159 2013-02-05 21:32
/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/User/.staging/job_201302050715_0032/job.jar

Thanks,
On Fri, Apr 19, 2013 at 2:51 PM, Bejoy Ks <[EMAIL PROTECTED]> wrote:

> This basically happens while running a mapreduce job. When a map reduce
> job is triggered the job files are put in hdfs with high replication (
> replication is controlled by  - 'mapred.submit.replication'  default value
> is 10).
>
> The job files are cleaned up after the job is completed and hence that
> could be the reason you are seeing the hdfs file system status as healthy
> after running the job.
>
>
> On Fri, Apr 19, 2013 at 1:04 PM, Mohit Vadhera <
> [EMAIL PROTECTED]> wrote:
>
>> its one (1). Output is below.
>>
>> .......Status: HEALTHY
>>  Total size:    903709673179 B
>>  Total dirs:    2906
>>  Total files:   11110
>>  Total blocks (validated):      20906 (avg. block size 43227287 B)
>>  Minimally replicated blocks:   20906 (100.0 %)
>>  Over-replicated blocks:        0 (0.0 %)
>>  Under-replicated blocks:       248 (1.1862624 %)
>>  Mis-replicated blocks:         0 (0.0 %)
>>  Default replication factor:    1
>>  Average block replication:     1.0
>>  Corrupt blocks:                0
>>  Missing replicas:              2232 (9.646469 %)
>>  Number of data-nodes:          1
>>  Number of racks:               1
>> FSCK ended at Fri Apr 19 03:47:04 EDT 2013 in 2224 milliseconds
>>
>>
>> The filesystem under path '/' is HEALTHY
>>
>>
>>
>> On Fri, Apr 19, 2013 at 12:28 PM, S, Manoj <[EMAIL PROTECTED]> wrote:
>>
>>>  It means that some of your data blocks are not replicated as intended.
>>> What is the value of “dfs.replication” in your hadoop-site.xml file?****
>>>
>>> ** **
>>>
>>> Can you paste the output of ****
>>>
>>> ** **
>>>
>>> *bin/hadoop fsck /****
>>>
>>> ** **
>>>
>>> --****
>>>
>>> Manoj****
>>>
>>> ** **
>>>
>>> *From:* Mohit Vadhera [mailto:[EMAIL PROTECTED]]
>>> *Sent:* Friday, April 19, 2013 12:09 PM
>>> *To:* <[EMAIL PROTECTED]>
>>> *Subject:* log****
>>>
>>> ** **
>>>
>>> Can anybody let me know the meaning of the below log plz " Target
>>> Replicas is 10 but found 1 replica(s)." ?
>>>
>>> /var/lib/hadoop-hdfs/cache/mapred/mapred/staging/test_user/.staging/job_201302180313_0623/job.split:
>>> Under replicated
>>> BP-2091347308-172.20.3.119-1356632249303:blk_6297333561560198850_70720.
>>> Target
>>> Replicas is 10 but found 1 replica(s).
>>>
>>> ****
>>>
>>> Thanks,****
>>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB