Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> BinInterSedes comparator failing for tuple


Copy link to this message
-
Re: BinInterSedes comparator failing for tuple
My guess is this is my doing. Will tackle this tomorrow.

Sent from my iPhone

On Oct 7, 2012, at 8:15 PM, Prasanth J <[EMAIL PROTECTED]> wrote:

> Hello everyone
>
> I recently updated my patch(PIG-2831) to work on the latest trunk and noticed that new trunk breaks my code. My code uses secondary sort on a tuple inside a databag. The issue seems to be related to BinInterSedes comparator. I have pasted the stack trace for reference. Can anyone let me know if this is side-effect of some other patch that is committed recently?
>
> java.io.IOException: Spill failed
>    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
>    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
>    at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
>    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
>    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
>    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
>    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>    at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.RuntimeException: Unexpected data type 39 found in stream.
>    at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.getGeneralizedDataType(BinInterSedes.java:1166)
>    at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:911)
>    at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:730)
>    at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
>    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
>    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
>    at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
>    at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
>    at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
>    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
>    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
>
>
> Thanks
> -- Prasanth
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB