Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - BinInterSedes comparator failing for tuple


+
Prasanth J 2012-10-08, 03:15
+
Jon Coveney 2012-10-08, 03:46
+
Prasanth J 2012-10-08, 03:48
+
Jonathan Coveney 2012-10-08, 18:15
Copy link to this message
-
Re: BinInterSedes comparator failing for tuple
Prasanth J 2012-10-08, 19:20
Thanks for fixing this Jon. It works now.

Thanks
-- Prasanth

On Oct 8, 2012, at 2:15 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:

> Prashanth,
>
> The fix is in. Please let me know if you have any more issues
>
> Jon
>
> 2012/10/7 Prasanth J <[EMAIL PROTECTED]>
>
>> Thanks Jon.
>>
>> Thanks
>> -- Prasanth
>>
>> On Oct 7, 2012, at 11:46 PM, Jon Coveney <[EMAIL PROTECTED]> wrote:
>>
>>> My guess is this is my doing. Will tackle this tomorrow.
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 7, 2012, at 8:15 PM, Prasanth J <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> Hello everyone
>>>>
>>>> I recently updated my patch(PIG-2831) to work on the latest trunk and
>> noticed that new trunk breaks my code. My code uses secondary sort on a
>> tuple inside a databag. The issue seems to be related to BinInterSedes
>> comparator. I have pasted the stack trace for reference. Can anyone let me
>> know if this is side-effect of some other patch that is committed recently?
>>>>
>>>> java.io.IOException: Spill failed
>>>>  at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
>>>>  at
>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
>>>>  at
>> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>>>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
>>>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
>>>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
>>>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
>>>>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>  at java.security.AccessController.doPrivileged(Native Method)
>>>>  at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>  at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>>>>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>> Caused by: java.lang.RuntimeException: Unexpected data type 39 found in
>> stream.
>>>>  at
>> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.getGeneralizedDataType(BinInterSedes.java:1166)
>>>>  at
>> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:911)
>>>>  at
>> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:730)
>>>>  at
>> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
>>>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
>>>>  at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
>>>>  at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
>>>>  at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
>>>>  at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>>>>  at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
>>>>  at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
>>>>  at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
>>>>
>>>>
>>>> Thanks
>>>> -- Prasanth
>>>>
>>
>>