Is it there is not enough space to keep the intermediate files?

How to find space allocated for HDFS and normal FS for a particular node
but overall the cluster is having more free space.

Cheers!
Manoj.

On Fri, Dec 7, 2012 at 11:22 AM, Marcos Ortiz <[EMAIL PROTECTED]> wrote:

> It seems that you don´t have enough space in your TaskTracker/DataNode
> node.
> Did you check available space on your dedicated hard drives to host your
> data in your TT/DN machine?
>
>
>
> On Fri 07 Dec 2012 12:38:27 AM CST, Manoj Babu wrote:
>
>> Hi All,
>>
>> I am getting the exception as below but the job continues to running
>> how to avoid this and whats the cause for the issue?
>>
>> 12/12/06 23:30:56 INFO mapred.JobClient: Task Id :
>> attempt_201211120903_7753_m_**000177_0, Status : FAILED
>> FSError: java.io.IOException: No space left on device
>> java.lang.Throwable: Child Error
>>         at org.apache.hadoop.mapred.**TaskRunner.run(TaskRunner.**
>> java:242)
>> Caused by: java.io.IOException: Task process exit with nonzero status
>> of -1.
>>         at org.apache.hadoop.mapred.**TaskRunner.run(TaskRunner.**
>> java:229)
>>
>> Cheers!
>> Manoj.
>>
>>
> --
> Marcos Luis Ortíz Valmaseda
> about.me/marcosortiz <http://about.me/marcosortiz>
> @marcosluis2186 <http://twitter.com/**marcosluis2186<http://twitter.com/marcosluis2186>
> >
>
>
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
> INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/**universidad.uci<http://www.facebook.com/universidad.uci>
> http://www.flickr.com/photos/**universidad_uci<http://www.flickr.com/photos/universidad_uci>
>

Seems you need to cleanup temporary files on your local filesystem.
Check dir /tmp/<user>/
Works for me :)

Thanks,
Abhijeet
On 7 Dec 2012 12:47, "Manoj Babu" <[EMAIL PROTECTED]> wrote:

> Is it there is not enough space to keep the intermediate files?
>
> How to find space allocated for HDFS and normal FS for a particular node
> but overall the cluster is having more free space.
>
> Cheers!
> Manoj.
>
>
>
> On Fri, Dec 7, 2012 at 11:22 AM, Marcos Ortiz <[EMAIL PROTECTED]> wrote:
>
>> It seems that you don´t have enough space in your TaskTracker/DataNode
>> node.
>> Did you check available space on your dedicated hard drives to host your
>> data in your TT/DN machine?
>>
>>
>>
>> On Fri 07 Dec 2012 12:38:27 AM CST, Manoj Babu wrote:
>>
>>> Hi All,
>>>
>>> I am getting the exception as below but the job continues to running
>>> how to avoid this and whats the cause for the issue?
>>>
>>> 12/12/06 23:30:56 INFO mapred.JobClient: Task Id :
>>> attempt_201211120903_7753_m_**000177_0, Status : FAILED
>>> FSError: java.io.IOException: No space left on device
>>> java.lang.Throwable: Child Error
>>>         at org.apache.hadoop.mapred.**TaskRunner.run(TaskRunner.**
>>> java:242)
>>> Caused by: java.io.IOException: Task process exit with nonzero status
>>> of -1.
>>>         at org.apache.hadoop.mapred.**TaskRunner.run(TaskRunner.**
>>> java:229)
>>>
>>> Cheers!
>>> Manoj.
>>>
>>>
>> --
>> Marcos Luis Ortíz Valmaseda
>> about.me/marcosortiz <http://about.me/marcosortiz>
>> @marcosluis2186 <http://twitter.com/**marcosluis2186<http://twitter.com/marcosluis2186>
>> >
>>
>>
>>
>> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>> INFORMATICAS...
>> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>>
>> http://www.uci.cu
>> http://www.facebook.com/**universidad.uci<http://www.facebook.com/universidad.uci>
>> http://www.flickr.com/photos/**universidad_uci<http://www.flickr.com/photos/universidad_uci>
>>
>
>

Look into the Hadoop logs directory as well
I ran into a similar situation some time back and found that one log file was the culprit

From: abhijeet gaikwad [mailto:[EMAIL PROTECTED]]
Sent: Friday, December 07, 2012 1:02 PM
To: [EMAIL PROTECTED]
Cc: Marcos Ortiz
Subject: Re: Reg: No space left on device Exception
Seems you need to cleanup temporary files on your local filesystem.
Check dir /tmp/<user>/
Works for me :)

Thanks,
Abhijeet
On 7 Dec 2012 12:47, "Manoj Babu" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Is it there is not enough space to keep the intermediate files?

How to find space allocated for HDFS and normal FS for a particular node but overall the cluster is having more free space.

Cheers!
Manoj.
On Fri, Dec 7, 2012 at 11:22 AM, Marcos Ortiz <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
It seems that you don´t have enough space in your TaskTracker/DataNode node.
Did you check available space on your dedicated hard drives to host your
data in your TT/DN machine?

On Fri 07 Dec 2012 12:38:27 AM CST, Manoj Babu wrote:
Hi All,

I am getting the exception as below but the job continues to running
how to avoid this and whats the cause for the issue?

12/12/06 23:30:56 INFO mapred.JobClient: Task Id :
attempt_201211120903_7753_m_000177_0, Status : FAILED
FSError: java.io.IOException: No space left on device
java.lang.Throwable: Child Error
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.io.IOException: Task process exit with nonzero status
of -1.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)

Cheers!
Manoj.

--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz<http://about.me/marcosortiz> <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are not
to copy, disclose, or distribute this e-mail or its contents to any other person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any damage
you may sustain as a result of any virus in this e-mail. You should carry out your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***

Thank you all for the inputs let me check those things.

Cheers!
Manoj.

On Fri, Dec 7, 2012 at 1:17 PM, Utkarsh Gupta <[EMAIL PROTECTED]>wrote:

>  Look into the Hadoop logs directory as well****
>
> I ran into a similar situation some time back and found that one log file
> was the culprit ****
>
> ** **
>
> *From:* abhijeet gaikwad [mailto:[EMAIL PROTECTED]]
> *Sent:* Friday, December 07, 2012 1:02 PM
> *To:* [EMAIL PROTECTED]
> *Cc:* Marcos Ortiz
> *Subject:* Re: Reg: No space left on device Exception****
>
> ** **
>
> Seems you need to cleanup temporary files on your local filesystem.
> Check dir /tmp/<user>/
> Works for me :)****
>
> Thanks,
> Abhijeet****
>
> On 7 Dec 2012 12:47, "Manoj Babu" <[EMAIL PROTECTED]> wrote:****
>
> Is it there is not enough space to keep the intermediate files?****
>
> ** **
>
> How to find space allocated for HDFS and normal FS for a particular node
> but overall the cluster is having more free space.****
>
>
> Cheers!****
>
> Manoj.****
>
>
>
> ****
>
> On Fri, Dec 7, 2012 at 11:22 AM, Marcos Ortiz <[EMAIL PROTECTED]> wrote:****
>
> It seems that you don´t have enough space in your TaskTracker/DataNode
> node.
> Did you check available space on your dedicated hard drives to host your
> data in your TT/DN machine?****
>
>
>
>
> On Fri 07 Dec 2012 12:38:27 AM CST, Manoj Babu wrote:****
>
> Hi All,
>
> I am getting the exception as below but the job continues to running
> how to avoid this and whats the cause for the issue?
>
> 12/12/06 23:30:56 INFO mapred.JobClient: Task Id :
> attempt_201211120903_7753_m_000177_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> Caused by: java.io.IOException: Task process exit with nonzero status
> of -1.
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>
> Cheers!
> Manoj.****
>
> ** **
>
> --
> Marcos Luis Ortíz Valmaseda
> about.me/marcosortiz <http://about.me/marcosortiz>
> @marcosluis2186 <http://twitter.com/marcosluis2186>
>
>
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
> INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/universidad.uci
> http://www.flickr.com/photos/universidad_uci****
>
> ** **
>
> **************** CAUTION - Disclaimer *****************
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
> for the use of the addressee(s). If you are not the intended recipient, please
> notify the sender by e-mail and delete the original message. Further, you are not
> to copy, disclose, or distribute this e-mail or its contents to any other person and
> any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
> every reasonable precaution to minimize this risk, but is not liable for any damage
> you may sustain as a result of any virus in this e-mail. You should carry out your
> own virus checks before opening the e-mail or attachment. Infosys reserves the
> right to monitor and review the content of all messages sent to or from this e-mail
> address. Messages sent to or from this e-mail address may be stored on the
> Infosys e-mail system.
> ***INFOSYS******** End of Disclaimer ********INFOSYS***
>
>

Another tip...  Is this a 32-bit Linux system?  Check for any files that
have grown to 2GB.

Marco
On Fri, Dec 7, 2012 at 1:38 AM, Manoj Babu <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> I am getting the exception as below but the job continues to running how
> to avoid this and whats the cause for the issue?
>
> 12/12/06 23:30:56 INFO mapred.JobClient: Task Id :
> attempt_201211120903_7753_m_000177_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> Caused by: java.io.IOException: Task process exit with nonzero status of
> -1.
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>
> Cheers!
> Manoj.
>
>

Hi Macro,

It a 62-bit linux system. Let me check that too!

Cheers!
Manoj.

On Fri, Dec 7, 2012 at 5:09 PM, Marco Shaw <[EMAIL PROTECTED]> wrote:

> Another tip...  Is this a 32-bit Linux system?  Check for any files that
> have grown to 2GB.
>
> Marco
>
>
> On Fri, Dec 7, 2012 at 1:38 AM, Manoj Babu <[EMAIL PROTECTED]> wrote:
>
>> Hi All,
>>
>> I am getting the exception as below but the job continues to running how
>> to avoid this and whats the cause for the issue?
>>
>> 12/12/06 23:30:56 INFO mapred.JobClient: Task Id :
>> attempt_201211120903_7753_m_000177_0, Status : FAILED
>> FSError: java.io.IOException: No space left on device
>> java.lang.Throwable: Child Error
>>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> Caused by: java.io.IOException: Task process exit with nonzero status of
>> -1.
>>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>>
>> Cheers!
>> Manoj.
>>
>>
>

Hi Manoj

Go to the JT web UI, browse to the failed tasks. Identify which task threw the space related error. Ssh to that node and check the disk space on that node. Some partitions might have got 100% filled.

Regards
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: Manoj Babu <[EMAIL PROTECTED]>
Date: Fri, 7 Dec 2012 18:55:18
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Reg: No space left on device Exception

Hi Macro,

It a 62-bit linux system. Let me check that too!

Cheers!
Manoj.

On Fri, Dec 7, 2012 at 5:09 PM, Marco Shaw <[EMAIL PROTECTED]> wrote:

> Another tip...  Is this a 32-bit Linux system?  Check for any files that
> have grown to 2GB.
>
> Marco
>
>
> On Fri, Dec 7, 2012 at 1:38 AM, Manoj Babu <[EMAIL PROTECTED]> wrote:
>
>> Hi All,
>>
>> I am getting the exception as below but the job continues to running how
>> to avoid this and whats the cause for the issue?
>>
>> 12/12/06 23:30:56 INFO mapred.JobClient: Task Id :
>> attempt_201211120903_7753_m_000177_0, Status : FAILED
>> FSError: java.io.IOException: No space left on device
>> java.lang.Throwable: Child Error
>>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> Caused by: java.io.IOException: Task process exit with nonzero status of
>> -1.
>>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>>
>> Cheers!
>> Manoj.
>>
>>
>