Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - issue about running hive MR job in hadoop


Copy link to this message
-
Re: issue about running hive MR job in hadoop
ch huang 2014-01-03, 08:40
yes i checked the code ,and find the Exception from

     lfs.mkdir(userFileCacheDir, null, false);

also find the AM located in CHBM224 ,all will failed but,AM located in
CHBM223,all success

in CHBM224

# ls -l /data/mrlocal/1/yarn/
total 8
drwxrwxrwx 5 yarn yarn 4096 Nov  5 20:50 local
drwxr-xr-x 3 yarn yarn 4096 Jan  3 15:57 logs
# ls -l /data/mrlocal/2/yarn/
total 8
drwxrwxrwx 5 yarn yarn 4096 Nov  5 20:50 local
drwxr-xr-x 3 yarn yarn 4096 Jan  3 15:57 logs

in CHBM223

# ls /data/mrlocal/1/yarn/ -l
total 8
drwxr-xr-x 5 yarn yarn 4096 Nov  5 20:51 local
drwxr-xr-x 3 yarn yarn 4096 Jan  3 15:46 logs
# ls /data/mrlocal/2/yarn/ -l
total 8
drwxr-xr-x 5 yarn yarn 4096 Nov  5 20:51 local
drwxr-xr-x 3 yarn yarn 4096 Jan  3 15:46 logs
i also find if i let abnormal node (CHBM224) run and shutdown the other
normal node ,when i submit a MR job use hive ,and the dir
"/data/mrlocal/2/yarn/local/usercache/hive/filecache" ,it's mode will flush
to 710 ,even i change the file to 755,but i test on a normal node (open one
normal node ,and shutdown others) ,the dir mode will not changed
# ls -l /data/mrlocal/2/yarn/local/usercache/hive/
total 16
drwx--x---   7 yarn yarn  4096 Jan  3 16:30 appcache
drwx--x--- 148 yarn yarn 12288 Jan  3 10:03 filecache
On Fri, Jan 3, 2014 at 3:52 PM, Bing Jiang <[EMAIL PROTECTED]> wrote:

>  Could you check your yarn-local directory authority? From the diagnosis,
> error occurs at mkdir in local directory.
> I guess something wrong with  local direcotry which is set as yarn local
> dir.
>
>
>
> 2014/1/3 ch huang <[EMAIL PROTECTED]>
>
>> hi, i submit a MR job through hive ,but when it run stage-2 ,it failed
>> why?
>>
>>  it seems permission problem ,but i do not know which dir cause the
>> problem
>>
>> Application application_1388730279827_0035 failed 1 times due to AM
>> Container for appattempt_1388730279827_0035_000001 exited with exitCode:
>> -1000 due to: EPERM: Operation not permitted at
>> org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) at
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:581)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:388)
>> at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1041) at
>> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
>> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:190) at
>> org.apache.hadoop.fs.FileContext$4.next(FileContext.java:698) at
>> org.apache.hadoop.fs.FileContext$4.next(FileContext.java:695) at
>> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
>> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:695) at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:385)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:130)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:861)
>> .Failing this attempt.. Failing the application.
>>
>
>
>
> --
> Bing Jiang
> Tel:(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>