Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> RE: Permission problem


Copy link to this message
-
Re: Permission problem
It looks like hadoop.tmp.dir is being used both for local and hdfs directories. Can you create a jira for this?

What i recommended is that you create /data/hadoop/tmp on hdfs and chmod -R /data

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On Apr 30, 2013, at 8:22 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:

> I am not clear on what you are suggesting to create on HDFS or the local file system. As I understand it hadoop.tmp.dir is the local file system. I changed it so that the temporary files would be on a disk that has more capacity then /tmp. So you are suggesting that I create /data/hadoop/tmp on HDFS. I already have this created.
>  
> Found 1 items
> drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred
> kevin@devUbuntu05:/etc/hadoop/conf$ hadoop fs -ls -d /tmp
> Found 1 items
> drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp
>  
> When you suggest that I ‘chmod –R 777 /data’. You are suggesting that I open up all the data to everyone? Isn’t that a bit extreme? First /data is the mount point for this drive and there are other uses for this drive than hadoop so there are other folders. That is why there is /data/hadoop. As far as hadoop is concerned:
>  
> kevin@devUbuntu05:/etc/hadoop/conf$ ls -l /data/hadoop/
> total 12
> drwxrwxr-x 4 hdfs   hadoop 4096 Apr 29 16:38 dfs
> drwxrwxr-x 3 mapred hadoop 4096 Apr 29 11:33 mapred
> drwxrwxrwx 3 hdfs   hadoop 4096 Apr 19 15:14 tmp
>  
> dfs would be where the data blocks for the hdfs file system would go, mapred would be the folder for M/R jobs, and tmp would be temporary storage. These are all on the local file system. Do I have to make all of this read-write for everyone in order to get it to work?
>  
> From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 10:01 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Permission problem
>  
> ah
>  
> this is what mapred.sytem.dir defaults to
>  
> <property>
>   <name>mapred.system.dir</name>
>   <value>${hadoop.tmp.dir}/mapred/system</value>
>   <description>The directory where MapReduce stores control files.
>   </description>
> </property>
>  
>  
> So thats why its trying to write to /data/hadoop/tmp/hadoop-mapred/mapred/system
>  
>  
> So if you want hadoop.tmp.dir to be /data/hadoop/tmp/hadoop-${user.name}  then i would suggest that create /data/hadoop/tmp on hdfs and chmod -R 777 /data or you can remove the hadoop.tmp.dir from your configs and let it be set to the default value of
>  
> <property>
>   <name>hadoop.tmp.dir</name>
>   <value>/tmp/hadoop-${user.name}</value>
>   <description>A base for other temporary directories.</description>
> </property>
>  
> So to fix your problem you can do the above or set mapred.system.dir to /tmp/mapred/system in your mapred-site.xml.
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>  
> On Apr 30, 2013, at 7:55 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:
>
>
> In core-site.xml I have:
>  
> <property>
>   <name>fs.default.name</name>
>   <value>hdfs://devubuntu05:9000</value>
>   <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. </description>
> </property>
>  
> In hdfs-site.xml I have
>  
> <property>
>   <name>hadoop.tmp.dir</name>
>   <value>/data/hadoop/tmp/hadoop-${user.name}</value>
>   <description>Hadoop temporary folder</description>
> </property>
>  
>  
> From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 9:48 AM
> To: Kevin Burton
> Cc: [EMAIL PROTECTED]
> Subject: Re: Permission problem
>  
> Based on the logs your system dir is set to
>  
> hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system
>  
>  
> what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?
>  
>  
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>  
> On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB