Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Problem with MR job


+
George Kousiouris 2011-09-21, 13:58
+
Harsh J 2011-09-21, 14:06
+
Uma Maheswara Rao G 72686... 2011-09-21, 14:08
+
George Kousiouris 2011-09-21, 14:15
Copy link to this message
-
Re: Problem with MR job
George Kousiouris 2011-09-21, 14:35

Hi,

Some more logs, specifically from the JobTracker:

2011-09-21 10:22:43,482 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201109211018_0001
2011-09-21 10:22:43,538 ERROR org.apache.hadoop.mapred.JobHistory:
Failed creating job history log file for job job_201109211018_0001
java.io.FileNotFoundException:
/usr/lib/hadoop-0.20/logs/history/master_1316614721548_job_201109211018_0001_hdfs_Input+Driver+running+over+input%3A+hdfs%3A%2F%2Fmaster%2Fuse
(P$
         at java.io.FileOutputStream.open(Native Method)
         at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
         at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:189)
         at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:185)
         at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:243)
         at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:336)
         at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:369)
         at
org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1223)
         at
org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:681)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:396)
         at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
         at
org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:678)
         at
org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4013)
         at
org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:662)
2011-09-21 10:22:43,666 ERROR org.apache.hadoop.mapred.JobHistory:
Failed to store job conf in the log dir
java.io.FileNotFoundException:
/usr/lib/hadoop-0.20/logs/history/master_1316614721548_job_201109211018_0001_conf.xml
(Permission denied)
         at java.io.FileOutputStream.open(Native Method)
         at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
         at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:189)
         at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:185)
         at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:243)
         at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:336)
         at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:369)
On 9/21/2011 5:15 PM, George Kousiouris wrote:
>
> Hi,
>
> The status seems healthy and the datanodes live:
> Status: HEALTHY
>  Total size:    118805326 B
>  Total dirs:    31
>  Total files:    38
>  Total blocks (validated):    38 (avg. block size 3126455 B)
>  Minimally replicated blocks:    38 (100.0 %)
>  Over-replicated blocks:    0 (0.0 %)
>  Under-replicated blocks:    9 (23.68421 %)
>  Mis-replicated blocks:        0 (0.0 %)
>  Default replication factor:    1
>  Average block replication:    1.2368422
>  Corrupt blocks:        0
>  Missing replicas:        72 (153.19148 %)
>  Number of data-nodes:        2
>  Number of racks:        1
> FSCK ended at Wed Sep 21 10:06:17 EDT 2011 in 9 milliseconds
>
>
> The filesystem under path '/' is HEALTHY
>
> The jps command has the following output:
> hdfs@master:~$ jps
> 24292 SecondaryNameNode
> 30010 Jps
> 24109 DataNode
> 23962 NameNode
>
> Shouldn't this have two datanode listings? In our system, one of the
> datanodes and the namenode is the same machine, but i seem to remember
> that in the past even with this setup two datanode listings appeared
George Kousiouris
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: [EMAIL PROTECTED]
Site: http://users.ntua.gr/gkousiou/

National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
+
Uma Maheswara Rao G 72686... 2011-09-21, 15:40