Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Where does Hadoop store its maps?


Copy link to this message
-
Re: Where does Hadoop store its maps?
Thank you, Harsh and Madhu, that is exactly what I was looking for.

Mark

On Tue, May 22, 2012 at 8:36 AM, madhu phatak <[EMAIL PROTECTED]> wrote:

> Hi,
>  Set "mapred.local.dir" in mapred-site.xml to point a directory on /mnt so
> that it will not use ec2 instance EBS.
>
> On Tue, May 22, 2012 at 6:58 PM, Mark Kerzner <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,
> >
> > I am using a Hadoop cluster of my own construction on EC2, and I am
> running
> > out of hard drive space with maps. If I knew which directories are used
> by
> > Hadoop for map spill, I could use the large ephemeral drive on EC2
> machines
> > for that. Otherwise, I would have to keep increasing my available hard
> > drive on root, and that's not very smart.
> >
> > Thank you. The error I get is below.
> >
> > Sincerely,
> > Mark
> >
> >
> >
> > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> > any valid local directory for output/file.out
> >        at
> >
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:376)
> >        at
> >
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
> >        at
> >
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127)
> >        at
> >
> org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:69)
> >        at
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1495)
> >        at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1180)
> >        at
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:582)
> >        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:649)
> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> >        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at javax.security.auth.Subject.doAs
> > java.io.IOException: Spill failed
> >        at
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:886)
> >        at
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
> >        at
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >        at
> > org.freeeed.main.ZipFileProcessor.emitAsMap(ZipFileProcessor.java:279)
> >        at
> >
> org.freeeed.main.ZipFileProcessor.processWithTrueZip(ZipFileProcessor.java:107)
> >        at
> > org.freeeed.main.ZipFileProcessor.process(ZipFileProcessor.java:55)
> >        at org.freeeed.main.Map.map(Map.java:70)
> >        at org.freeeed.main.Map.map(Map.java:24)
> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> >        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at javax.security.auth.Subject.doAs(Subject.java:415)
> >        at org.apache.hadoop.security.UserGroupInformation.doAs(User
> > java.io.IOException: Spill failed
> >        at
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:886)
> >        at
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
> >        at
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >        at
> > org.freeeed.main.ZipFileProcessor.emitAsMap(ZipFileProcessor.java:279)
> >        at
> >
> org.freeeed.main.ZipFileProcessor.processWithTrueZip(ZipFileProcessor.java:107)
> >        at
> > org.freeeed.main.ZipFileProcessor.process(ZipFileProcessor.java:55)
> >        at org.freeeed.main.Map.map(Map.java:70)
> >        at org.freeeed.main.Map.map(Map.java:24)
> >        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)