Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Mappers crashing due to running out of heap space during initialisation


Copy link to this message
-
Re: Mappers crashing due to running out of heap space during initialisation
Lower io.sort.mb or raise the heap size for the task. -C

On Tue, Apr 26, 2011 at 10:55 AM, James Hammerton
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have a job that runs fine with a small data set in pseudo-distributed mode
> on my desktop workstation but when I run it on our Hadoop cluster it falls
> over, crashing during the initialisation of some of the mappers. The errors
> look like this:
>
> 2011-04-26 14:34:04,494 FATAL org.apache.hadoop.mapred.TaskTracker: Error
> running child : java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:743)
>
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:487)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> The mapper itself buffers only the previous input and the objects are small
> (max 201K in size, most under 50k), so I don't know why this is happening.
>
> What exactly is happening in the area of code referred to in the stack
> trace?
>
> Cheers,
>
> James
>
> --
> James Hammerton | Senior Data Mining Engineer
> www.mendeley.com/profiles/james-hammerton
>
> Mendeley Limited | London, UK | www.mendeley.com
> Registered in England and Wales | Company Number 6419015
>
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB