Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> What can cause a spill to fail?


Copy link to this message
-
Re: What can cause a spill to fail?
I see that you are running in local mode. You might want to specify a
-Xmx value if aren't doing that, the default max heap size is usually
quite low.

Is the udf that you are using holding onto lot of data ? Does this
happen with other builtin udfs as well ?

-Thejas

On 10/17/11 1:19 AM, Jonathan Coveney wrote:
> I got some more info. The random "INITIAL whatever" is just me testing an
> algebraic udf in jruby. I'm pasting the stack trace from where the errors
> start.
>
> Exception in thread "Low Memory Detector" Exception in thread "Poller
> SunPKCS11-Darwin" java.lang.OutOfMemoryError: Java heap space
> INITIAL item 271346 becomes 271346
> INITIAL item 271346 becomes 271346
> INITIAL item 271347 becomes 271347
> INITIAL item 271347 becomes 271347
>      at sun.security.pkcs11.wrapper.PKCS11.C_GetSlotInfo(Native Method)
>      at sun.security.pkcs11.SunPKCS11.initToken(SunPKCS11.java:767)
>      at sun.security.pkcs11.SunPKCS11.access$100(SunPKCS11.java:42)
>      at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:700)
>      at java.lang.Thread.run(Thread.java:680)
> INITIAL item 271348 becomes 271348
> INITIAL item 271348 becomes 271348
> 2011-10-17 01:17:38,563 [communication thread] INFO
> org.apache.hadoop.mapred.LocalJobRunner -
> INITIAL item 271349 becomes 271349
> INITIAL item 271349 becomes 271349
> 2011-10-17 01:17:40,810 [SpillThread] FATAL
> org.apache.hadoop.mapred.TaskRunner - Task attempt_local_0001_m_000000_0
> failed : java.lang.OutOfMemoryError: Java heap space
>      at java.lang.Long.valueOf(Long.java:557)
>      at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:297)
>      at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:251)
>      at org.apache.pig.data.BinInterSedes.readTuple(BinInterSedes.java:111)
>      at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:270)
>      at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:251)
>      at
> org.apache.pig.data.BinInterSedes.addColsToTuple(BinInterSedes.java:556)
>      at org.apache.pig.data.BinSedesTuple.readFields(BinSedesTuple.java:64)
>      at
> org.apache.pig.impl.io.PigNullableWritable.readFields(PigNullableWritable.java:114)
>      at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
>      at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
>      at
> org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
>      at
> org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(ReduceContext.java:163)
>      at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POCombinerPackage.getNext(POCombinerPackage.java:141)
>      at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.processOnePackageOutput(PigCombiner.java:171)
>      at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:162)
>      at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:1)
>      at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>      at
> org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1222)
>      at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1265)
>      at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:686)
>      at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1173)
>
> 2011-10-17 01:17:40,811 [SpillThread] FATAL
> org.apache.hadoop.mapred.LocalJobRunner - Fatal: java.lang.OutOfMemoryError:
> Java heap space
>      at java.lang.Long.valueOf(Long.java:557)
>      at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:297)
>      at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:251)
>      at org.apache.pig.data.BinInterSedes.readTuple(BinInterSedes.java:111)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB