Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> In the constructor of JobInProgress, why is it safe to call FileSyste.closeAllForUGI().


Copy link to this message
-
In the constructor of JobInProgress, why is it safe to call FileSyste.closeAllForUGI().
Hi,

This might be a naive question, but I am having a difficult time to
understand it. At the end of the constructor of JobInProgress, in the
finally clause, the code calls
FileSystem.closeAllForUGI(UserGroupInformation.getCurrentUser()), but why
is it safe.

My concern is that the current user is the owner of jobtracker, so it will
close all the files the jobtracker is writing, such as a jobtoken file
another jip is currently writing.

I modified some of the code of hadoop-1.1.0 for my research project and saw
the following error. It could be some bug in my code, but I suspect it is a
combined effect of this closeAllForUGI function and perhaps a race
condition in the DFSClient$LeaseChecker.close().

Could you help me understand why it is safe to call this
FileSystem.closeAllForUGI function at the end of the JobInProgress
constructor?

Thank you very much indeed.

==
2013-03-21 21:04:56,677 INFO org.apache.hadoop.mapred.JobTracker:
Initializing job_201303212104_0005
2013-03-21 21:04:56,677 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201303212104_0005
2013-03-21 21:04:56,841 INFO org.apache.hadoop.hdfs.DFSClient:
read.type=dfs block=blk_700146088908855679_2191
src=/home/ec2-user/Hadoop/tmp/mapred/staging/ec2-user/.staging/job_201303212104_0006/job.xml
2013-03-21 21:04:56,897 INFO org.apache.hadoop.hdfs.DFSClient: ===streamer
closed here===:org.apache.hadoop.hdfs.DFSClient$DFSOutputStream@239cd5f5dfsclientDFSClient[clientName=DFSClient_-536745704, ugi=ec2-user]
leasechecker remove source:
/test/out/intrecord/4/_logs/history/job_201303212104_0005_conf.xml
2013-03-21 21:04:56,898 INFO org.apache.hadoop.mapred.JobInProgress:
===generating job token==2013-03-21 21:04:56,931 INFO org.apache.hadoop.mapred.JobInProgress:
job_201303212104_0006: nMaps=1 nReduces=0 max=-1
2013-03-21 21:04:56,932 INFO org.apache.hadoop.mapred.JobInProgress:
===closeAllForUGI here===:ec2-user jobid= job_201303212104_0006
*2013-03-21 21:04:56,973 INFO org.apache.hadoop.hdfs.DFSClient: ===streamer
closed here===:org.apache.hadoop.hdfs.DFSClient$DFSOutputStream@6d56d7c8dfsclientDFSClient[clientName=DFSClient_1735333485, ugi=ec2-user]
leasechecker remove source:
/home/ec2-user/Hadoop/tmp/mapred/system/job_201303212104_0005/jobToken*
2013-03-21 21:04:56,973 INFO org.apache.hadoop.mapred.JobInProgress:
jobToken generated and stored with users keys in
/home/ec2-user/Hadoop/tmp/mapred/system/job_201303212104_0005/jobToken
*2013-03-21 21:04:56,973 INFO org.apache.hadoop.hdfs.DFSClient: ===streamer
null to close===:org.apache.hadoop.hdfs.DFSClient$DFSOutputStream@6d56d7c8dfsclientDFSClient[clientName=DFSClient_1735333485, ugi=ec2-user]
leasechecker should remove source:
/home/ec2-user/Hadoop/tmp/mapred/system/job_201303212104_0005/jobToken*
2013-03-21 21:04:56,974 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:ec2-user cause:java.io.IOException: java.lang.NullPointerException
*2013-03-21 21:04:56,976 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 9101, call submitJob(job_201303212104_0006, hdfs://
10.151.93.195:9100/home/ec2-user/Hadoop/tmp/mapred/staging/ec2-user/.staging/job_201303212104_0006,
org.apache.hadoop.security.Credentials@6070c38c) from 10.151.93.195:41602:
error: java.io.IOException: java.lang.NullPointerException*
java.io.IOException: java.lang.NullPointerException
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3799)
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3763)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
Caused by: java.lang.NullPointerException
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeThreads(DFSClient.java:4177)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:4220)
    at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:4145)
    at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1475)
    at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:341)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:339)
    at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1499)
    at org.apache.hadoop.fs.FileSystem.closeAllForUGI(FileSystem.java:289)
    at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:498)
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3797)
    ... 12 more
2013-03-21 21:04:56,979 INFO org.apache.hadoop.hdfs.DFSClient:
read.type=dfs block=blk_972265969568196888_2183
src=/home/ec2-user/Hadoop/tmp/mapred/staging/ec2-user/.staging/job_201303212104_0005/job.splitmetainfo
Xiao Yu