This is the problem:

drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..

Your /data/1 directory seems to be owned by hdfs and restricted only
to it (700). I'm not sure this is necessary and you can perhaps make
it 755 at least.

Or perhaps what you may have is a misconfig wherein you've set your DN
config of dfs.datanode.data.dir to a set of /data/{1..n} instead of
the regular /data/{1..n}/dn or so. The DN has a feature where it
applies dfs.datanode.data.dir.perm (700) to its configured disks for
security - thereby marking all of your mount entry points as 700 and
causing trouble with the mapred.local.dir config of
/data/{1..n}/mapred which resides under it.

On Wed, Jan 23, 2013 at 10:58 PM, Corbett Martin <[EMAIL PROTECTED]> wrote:
> Question
>
>
>
> We’re trying out Cloudera Manager and CDH4 in a Clustered deployment and
> having trouble getting the Task Trackers to start up.
>
>
>
> The error says (full stacktrace below)
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
>      at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
>      at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
>      at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> What directories is it referring to?  We have the following specified on the
> local file system.
>
>
>
> ls -al /data/1/mapred/
>
> drwxr-xr-x 4 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/2/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/3/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> When we try to start the Map Reduce service from Cloudera Manager the Task
> Trackers fail to start.  The Job Tracker starts fine.  Here’s the error from
> the log file.
>
>
>
> ************************************************************/
>
> 2013-01-23 10:48:35,951 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
>
> /************************************************************
>
> STARTUP_MSG: Starting TaskTracker
>
> STARTUP_MSG:   host = foo-dev-arch-02/10.204.12.63
>
> STARTUP_MSG:   args = []
>
> STARTUP_MSG:   version = 2.0.0-mr1-cdh4.1.1
>
> STARTUP_MSG:   classpath > /var/run/cloudera-scm-agent/process/499-mapreduce-TASKTRACKER:/usr/java/jdk1.6.0_30/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2