Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Re: Hadoop NON DFS space


+
Jagat Singh 2013-01-16, 07:58
Copy link to this message
-
Re: Hadoop NON DFS space
Here is my problem
I am using bulk loading for Hbase using MapReduce Program

Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %

But when i run my program

Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %

I have disable WAL in hbase  still its consuming non-dfs
and  my program fails have tried lot times but no luck

SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE

I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
large extent
13/01/17 08:44:07 INFO mapred.JobClient:  map 83% reduce 22%
13/01/17 08:44:09 INFO mapred.JobClient:  map 84% reduce 22%
13/01/17 08:44:12 INFO mapred.JobClient:  map 85% reduce 22%
13/01/17 08:44:15 INFO mapred.JobClient:  map 86% reduce 22%
13/01/17 08:44:18 INFO mapred.JobClient:  map 87% reduce 22%
13/01/17 08:44:22 INFO mapred.JobClient:  map 79% reduce 22%
13/01/17 08:44:25 INFO mapred.JobClient:  map 80% reduce 25%
13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
attempt_201301170837_0004_m_000009_0, Status : FAILED
FSError: java.io.IOException: No space left on device
java.lang.Throwable: Child Error
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of
/tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
failed.
        at
org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
        at
org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
        at
org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
13/01/17 08:44:28 INFO mapred.JobClient:  map 82% reduce 25%
13/01/17 08:44:31 INFO mapred.JobClient:  map 83% reduce 25%
13/01/17 08:45:07 INFO mapred.JobClient:  map 83% reduce 27%
On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> I think you can still run with the OS on another drive, or on a live
> USB drive, or even on the memory only, loaded from the network while
> the server is booting from the network drive, etc. No?
>
> JM
>
> 2013/1/16, Mohammad Tariq <[EMAIL PROTECTED]>:
> > That would be really cool Chris.
> > +1 for that.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <[EMAIL PROTECTED]> wrote:
> >
> >> Ha, you joke, but we're planning on running with no local OS.  If it
> >> works
> >> as planned I'll post a nice summary of our approach. :)
> >>
> >>
> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >>> <kidding> Wipe your OS out. </kidding>
> >>>
> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
> >>>
> >>>
> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
> >>> <[EMAIL PROTECTED]>wrote:
> >>>
> >>>>
> >>>> how to remove non dfs space from  hadoop cluster
> >>>>
> >>>> --
> >>>> *
> >>>> *
> >>>> *
> >>>>
> >>>> Thanx and Regards*
> >>>> * Vikas Jadhav*
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>>
> >>
> >>
> >
>

--
*
*
*

Thanx and Regards*
* Vikas Jadhav*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB