Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> BinInterSedes comparator failing for tuple


+
Prasanth J 2012-10-08, 03:15
+
Jon Coveney 2012-10-08, 03:46
+
Prasanth J 2012-10-08, 03:48
Copy link to this message
-
Re: BinInterSedes comparator failing for tuple
Prashanth,

The fix is in. Please let me know if you have any more issues

Jon

2012/10/7 Prasanth J <[EMAIL PROTECTED]>

> Thanks Jon.
>
> Thanks
> -- Prasanth
>
> On Oct 7, 2012, at 11:46 PM, Jon Coveney <[EMAIL PROTECTED]> wrote:
>
> > My guess is this is my doing. Will tackle this tomorrow.
> >
> > Sent from my iPhone
> >
> > On Oct 7, 2012, at 8:15 PM, Prasanth J <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hello everyone
> >>
> >> I recently updated my patch(PIG-2831) to work on the latest trunk and
> noticed that new trunk breaks my code. My code uses secondary sort on a
> tuple inside a databag. The issue seems to be related to BinInterSedes
> comparator. I have pasted the stack trace for reference. Can anyone let me
> know if this is side-effect of some other patch that is committed recently?
> >>
> >> java.io.IOException: Spill failed
> >>   at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
> >>   at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> >>   at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >>   at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
> >>   at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
> >>   at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
> >>   at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> >>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> >>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> >>   at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >>   at java.security.AccessController.doPrivileged(Native Method)
> >>   at javax.security.auth.Subject.doAs(Subject.java:396)
> >>   at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
> >>   at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >> Caused by: java.lang.RuntimeException: Unexpected data type 39 found in
> stream.
> >>   at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.getGeneralizedDataType(BinInterSedes.java:1166)
> >>   at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:911)
> >>   at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:730)
> >>   at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
> >>   at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
> >>   at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
> >>   at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
> >>   at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
> >>   at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> >>   at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
> >>   at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
> >>   at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
> >>
> >>
> >> Thanks
> >> -- Prasanth
> >>
>
>
+
Prasanth J 2012-10-08, 19:20
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB