Hi all,

I have a job fail this morning because of 2 tasks were trying to write
into disk that somehow turned read-only.
Originally, i was thinking/dreaming that in this case somehow those 2
tasks will be exported automatically
to other dn/tt that  also has the required data block, and won't fail.

I strongly believe that Hadoop can do that but i just didn't know it
well enough to enable it.

/dev/sdj1 /hadoop10 ext3 ro,noatime,data=ordered 0 0

Error initializing attempt_201203211854_2633_m_000017_0: EROFS:
Read-only file system at
org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:496)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
at org.apache.hadoop.mapred.JobLocalizer.createLocalDirs(JobLocalizer.java:144)
at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:190)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1199)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1174)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1089)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2257)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2221)

Hope this make sense.
Patai

Hi Patai
     JobTracker automatically handles this situation by attempting the task
on different nodes.Could you verify the number of attempts that these
failed tasks made. Was that just one? If more whether all the
task attempts were triggered on the same node or not? Did all of them fail
with the same error? You can get this information from the jobtracker web
UI, drill down to task level and then further down a failed task.

Regards
Bejoy

On Thu, Mar 22, 2012 at 11:25 PM, Patai Sangbutsarakum <
[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I have a job fail this morning because of 2 tasks were trying to write
> into disk that somehow turned read-only.
> Originally, i was thinking/dreaming that in this case somehow those 2
> tasks will be exported automatically
> to other dn/tt that  also has the required data block, and won't fail.
>
> I strongly believe that Hadoop can do that but i just didn't know it
> well enough to enable it.
>
> /dev/sdj1 /hadoop10 ext3 ro,noatime,data=ordered 0 0
>
> Error initializing attempt_201203211854_2633_m_000017_0: EROFS:
> Read-only file system at
> org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) at
>
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:496)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
> at
> org.apache.hadoop.mapred.JobLocalizer.createLocalDirs(JobLocalizer.java:144)
> at
> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:190)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1199)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:396) at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at
> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1174)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1089)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2257)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2221)
>
> Hope this make sense.
> Patai
>