|
|
-
about pig-0.8.1 unit test: TestMergeJoinOuter
lulynn_2008 2011-08-23, 08:44
Hello Thejas Nair, During running ant -Dtestcase=TestMergeJoinOuter test with other JDK but not SUN JDK. I found my output is different from the one under SUN JDK: SUN JDK: passed Other JDK: failed with following information: Testcase: testCompilation took 0.443 sec FAILED junit.framework.AssertionFailedError: at org.apache.pig.test.TestMergeJoinOuter.testCompilation(TestMergeJoinOuter.java:116):assertTrue(oper.reducePlan.isEmpty());
I found the variable oper(in TestMergeJoinOuter.testCompilation) is different, here is the initialization of oper: LogicalPlan lp = lpt.buildPlan("C = join A by id left, B by id using 'merge';"); PigContext pc = new PigContext(ExecType.MAPREDUCE,cluster.getProperties()); pc.connect(); lp = lpt.buildPlan("store C into 'out';"); MROperPlan mrPlan = Util.buildMRPlan(Util.buildPhysicalPlan(lp, pc),pc); Iterator<MapReduceOper> itr = mrPlan.iterator(); MapReduceOper oper = itr.next(); Here are the output: SUN JDK: oper:(Name: MapReduce(-1,TestMapSideCogroup$DummyCollectableLoader,IsEmpty,PigStorage) - scope-39: Reduce Plan Empty | C: Store(hdfs://localhost.localdomain:33564/user/root/out:org.apache.pig.builtin.PigStorage) - scope-38 | | | |---C: New For Each(true,true)[tuple] - scope-37 | | | | | Project[bag][1] - scope-31 | | | | | POBinCond[bag] - scope-36 | | | | | |---Project[bag][2] - scope-32 | | | | | |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - scope-34 | | | | | | | |---Project[bag][2] - scope-33 | | | | | |---Constant({(,,)}) - scope-35 | | | |---C: MergeCogroup[tuple] - scope-30 | | | |---A: Load(hdfs://localhost.localdomain:33564/user/root/data1:org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader) - scope-22 Operator Key: scope-39)
Other JDK: oper:(Name: MapReduce(1) - scope-41: | Store(hdfs://localhost.localdomain:49160/tmp/temp-1456742965/tmp2077335416:org.apache.pig.impl.io.InterStorage) - scope-48 | | | |---New For Each(true)[bag] - scope-47 | | | | | Project[tuple][1] - scope-46 | | | |---Package[tuple]{tuple} - scope-45 | Local Rearrange[tuple]{tuple}(false) - scope-44 | | | | | Project[tuple][*] - scope-43 | | | |---Load(hdfs://localhost.localdomain:49160/user/root/data1:org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MergeJoinIndexer('org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader','kmonaaafhdhcaabdgkgbhggbcohfhegjgmcoebhchcgbhjemgjhdhehiibncbnjjmhgbjnadaaabejaaaehdgjhkgfhihaaaaaaaabhhaeaaaaaaakhdhcaaeogphcghcogbhagbgdgigfcohagjghcogcgbgdglgfgogecogigbgegpgphacogfhigfgdhfhegjgpgogfgoghgjgogfcohagihjhdgjgdgbgmemgbhjgfhccohagmgbgohdcofagihjhdgjgdgbgmfagmgbgoaaaaaaaaaaaaaaabacaaabfkaaangfgogeepggebgmgmejgohahfhehihcaacfgphcghcogbhagbgdgigfcohagjghcogjgnhagmcohagmgbgocoephagfhcgbhegphcfagmgbgoeaondlnechegpmlgacaaaiemaaakgneghcgpgnefgeghgfhdheaacdemgphcghcpgbhagbgdgigfcphagjghcpgjgnhagmcphfhegjgmcpenhfgmhegjengbhadlemaaafgnelgfhjhdheaaapemgkgbhggbcphfhegjgmcpengbhadlemaaahgnemgfgbhggfhdheaabaemgkgbhggbcphfhegjgmcpemgjhdhedlemaaaegnephahdhbaahoaaafemaaaggnfcgpgphehdhbaahoaaagemaaaognfdgpggheeghcgpgnefgeghgfhdhbaahoaaaeemaaamgnfdgpgghefegpefgeghgfhdhbaahoaaa eemaaaignfegpefgeghgfhdhbaahoaaaehihahdhcaacbgphcghcogbhagbgdgigfcohagjghcogjgnhagmcohfhegjgmcoenhfgmhegjengbhaaaaaaaaaaaaaaaacacaaabemaaaegnengbhahbaahoaaafhihahdhcaabbgkgbhggbcohfhegjgmcoeigbhdgiengbhaafahnkmbmdbgganbadaaacegaaakgmgpgbgeeggbgdhegphcejaaajhegihcgfhdgigpgmgehihadpeaaaaaaaaaaaamhhaiaaaaaabaaaaaaaaahihdhbaahoaaakdpeaaaaaaaaaaaamhhaiaaaaaabaaaaaaaabhdhcaacegphcghcogbhagbgdgigfcohagjghcogjgnhagmcohagmgbgocoephagfhcgbhegphcelgfhjaaaaaaaaaaaaaaabacaaacekaaacgjgeemaaafhdgdgphagfheaabcemgkgbhggbcpgmgbgoghcpfdhehcgjgoghdlhihaaaaaaaaaaaaaaablheaaafhdgdgphagfhdhcaafjgphcghcogbhagbgdgigfcohagjghcogcgbgdglgfgogecogigbgegpgphacogfhigfgdhfhegjgpgogfgoghgjgogfcohagihjhdgjgdgbgmemgbhjgfhccogfhihahcgfhdhdgjgpgoephagfhcgbhegphchdcofaepfahcgpgkgfgdheaaaaaaaaaaaaaaabacaaaffkaaakgphggfhcgmgpgbgegfgefkaabfhahcgpgdgfhdhdgjgoghecgbghepggfehfhagmgfhdfkaabehcgfhdhfgmhefdgjgoghgmgffehfhagmgfecgbghfkaaaehdhegbhcemaaahgdgpgmhfgngohdheaabfemgkgbhggbcphfhegjgmcpebhchcgbhjemgjhdhedlhihcaagc gphcghcogbhagbgdgigfcohagjghcogcgbgdglgfgogecogigbgegpgphacogfhigfgdhfhegjgpgogfgoghgjgogfcohagihjhdgjgdgbgmemgbhjgfhccogfhihahcgfhdhdgjgpgoephagfhcgbhegphchdcoefhihahcgfhdhdgjgpgoephagfhcgbhegphcaaaaaaaaaaaaaaabacaaabemaaadgmgpghheaacaemgphcghcpgbhagbgdgigfcpgdgpgngngpgohdcpgmgpghghgjgoghcpemgpghdlhihcaaemgphcghcogbhagbgdgigfcohagjghcogcgbgdglgfgogecogigbgegpgphacogfhigfgdhfhegjgpgogfgoghgjgogfcohagihjhdgjgdgbgmemgbhjgfhccofagihjhdgjgdgbgmephagfhcgbhegphcaaaaaaaaaaaaaaabacaaamfkaaafgbgdgdhfgnfkaaangjgohahfheebhehegbgdgigfgeejaabehcgfhbhfgfhdhegfgefagbhcgbgmgmgfgmgjhdgnecaaakhcgfhdhfgmhefehjhagfemaaafgbgmgjgbhdhbaahoaaaoemaaafgjgohahfheheaablemgphcghcpgbhagbgdgigfcphagjghcpgegbhegbcpfehfhagmgfdlemaaaggjgohahfhehdhbaahoaaagemaaangmgjgogfgbghgffehcgbgdgfhcheaachemgphcghcpgbhagbgdgigfcphagjghcphagfgocphfhegjgmcpemgjgogfgbghgffehcgbgdgfhcdlemaaadgmgpghhbaahoaabeemaaahgphfhehahfhehdhbaahoaaagemaaakhagbhcgfgohefagmgbgoheaafaemgphcghcpgbhagbgdgigfcphagjghcpgcgbgdglgfgogecpgigbgegpg phacpgfhigfgdhfhegjgpgogfgoghgjgogfcphagihjhdgjgdgbgmemgbhjgfhccphagmgbgohdcpfagihjhdgjgdgbgmfagmgbgodlemaaadhcgfhdheaaeeemgphcghcpgbhagbgdgigfcphagjghcpgcgbgdglgfgogecpgigbgegpgphacpgfhigfgdhfhegjgpgogfgoghgjgogfcphagihjhdgjgdgbgmemgbhjgfhccpfcgfhdhfgmhedlhihcaacbgphcghcogbhagbgdgigfcohagjghcogjgnhagmcohagmgbgocoephagfhcgbhegphcaaaaaaaaaaaaaaabacaaabemaaaegnelgfhjheaacgemgphcghcpgbhagbgdgigfcphagjghcpgjgnhagmcphagmgbgocpephagfhcgbhegphcelgfhjdlhihahbaahoaaapaaaappppppppdcheaaacgjgehahahahdhcaaclgphcghcogbhagbgdgigfcogdgpgngngpgohdcogmgpghghgjgoghcogjgnhagmcoemgpghdeekemgpghghgfhccikmpnoicknfncdiacaaabemaaaegogbgngf
-
Re: about pig-0.8.1 unit test: TestMergeJoinOuter
Ashutosh Chauhan 2011-08-25, 00:58
Hey Lulynn,
Plan certainly looks different. I am surprised it is because of different JDK. Unlikely, but a bug in your JDK ? Which other test cases, do you see fail with this JDK?
Ashutosh
2011/8/23 lulynn_2008 <[EMAIL PROTECTED]>
> Hello Thejas Nair, > During running ant -Dtestcase=TestMergeJoinOuter test with other JDK but > not SUN JDK. I found my output is different from the one under SUN JDK: > SUN JDK: passed > Other JDK: failed with following information: > Testcase: testCompilation took 0.443 sec > FAILED > junit.framework.AssertionFailedError: > at > org.apache.pig.test.TestMergeJoinOuter.testCompilation(TestMergeJoinOuter.java:116):assertTrue(oper.reducePlan.isEmpty()); > > I found the variable oper(in TestMergeJoinOuter.testCompilation) is > different, here is the initialization of oper: > LogicalPlan lp = lpt.buildPlan("C = join A by id left, B by id > using 'merge';"); > PigContext pc = new > PigContext(ExecType.MAPREDUCE,cluster.getProperties()); > pc.connect(); > lp = lpt.buildPlan("store C into 'out';"); > MROperPlan mrPlan = Util.buildMRPlan(Util.buildPhysicalPlan(lp, > pc),pc); > Iterator<MapReduceOper> itr = mrPlan.iterator(); > MapReduceOper oper = itr.next(); > Here are the output: > SUN JDK: > oper:(Name: > MapReduce(-1,TestMapSideCogroup$DummyCollectableLoader,IsEmpty,PigStorage) - > scope-39: > Reduce Plan Empty > | C: > Store(hdfs://localhost.localdomain:33564/user/root/out:org.apache.pig.builtin.PigStorage) > - scope-38 > | | > | |---C: New For Each(true,true)[tuple] - scope-37 > | | | > | | Project[bag][1] - scope-31 > | | | > | | POBinCond[bag] - scope-36 > | | | > | | |---Project[bag][2] - scope-32 > | | | > | | |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - > scope-34 > | | | | > | | | |---Project[bag][2] - scope-33 > | | | > | | |---Constant({(,,)}) - scope-35 > | | > | |---C: MergeCogroup[tuple] - scope-30 > | | > | |---A: > Load(hdfs://localhost.localdomain:33564/user/root/data1:org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader) > - scope-22 Operator Key: scope-39) > > Other JDK: > oper:(Name: MapReduce(1) - scope-41: > | > Store(hdfs://localhost.localdomain:49160/tmp/temp-1456742965/tmp2077335416:org.apache.pig.impl.io.InterStorage) > - scope-48 > | | > | |---New For Each(true)[bag] - scope-47 > | | | > | | Project[tuple][1] - scope-46 > | | > | |---Package[tuple]{tuple} - scope-45 > | Local Rearrange[tuple]{tuple}(false) - scope-44 > | | | > | | Project[tuple][*] - scope-43 > | | > | > |---Load(hdfs://localhost.localdomain:49160/user/root/data1:org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MergeJoinIndexer('org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader','kmonaaafhdhcaabdgkgbhggbcohfhegjgmcoebhchcgbhjemgjhdhehiibncbnjjmhgbjnadaaabejaaaehdgjhkgfhihaaaaaaaabhhaeaaaaaaakhdhcaaeogphcghcogbhagbgdgigfcohagjghcogcgbgdglgfgogecogigbgegpgphacogfhigfgdhfhegjgpgogfgoghgjgogfcohagihjhdgjgdgbgmemgbhjgfhccohagmgbgohdcofagihjhdgjgdgbgmfagmgbgoaaaaaaaaaaaaaaabacaaabfkaaangfgogeepggebgmgmejgohahfhehihcaacfgphcghcogbhagbgdgigfcohagjghcogjgnhagmcohagmgbgocoephagfhcgbhegphcfagmgbgoeaondlnechegpmlgacaaaiemaaakgneghcgpgnefgeghgfhdheaacdemgphcghcpgbhagbgdgigfcphagjghcpgjgnhagmcphfhegjgmcpenhfgmhegjengbhadlemaaafgnelgfhjhdheaaapemgkgbhggbcphfhegjgmcpengbhadlemaaahgnemgfgbhggfhdheaabaemgkgbhggbcphfhegjgmcpemgjhdhedlemaaaegnephahdhbaahoaaafemaaaggnfcgpgphehdhbaahoaaagemaaaognfdgpggheeghcgpgnefgeghgfhdhbaahoaaaeemaaamgnfdgpgghefegpefgeghgfhdhbaahoaaa > > eemaaaignfegpefgeghgfhdhbaahoaaaehihahdhcaacbgphcghcogbhagbgdgigfcohagjghcogjgnhagmcohfhegjgmcoenhfgmhegjengbhaaaaaaaaaaaaaaaacacaaabemaaaegnengbhahbaahoaaafhihahdhcaabbgkgbhggbcohfhegjgmcoeigbhdgiengbhaafahnkmbmdbgganbadaaacegaaakgmgpgbgeeggbgdhegphcejaaajhegihcgfhdgigpgmgehihadpeaaaaaaaaaaaamhhaiaaaaaabaaaaaaaaahihdhbaahoaaakdpeaaaaaaaaaaaamhhaiaaaaaabaaaaaaaabhdhcaacegphcghcogbhagbgdgigfcohagjghcogjgnhagmcohagmgbgocoephagfhcgbhegphcelgfhjaaaaaaaaaaaaaaabacaaacekaaacgjgeemaaafhdgdgphagfheaabcemgkgbhggbcpgmgbgoghcpfdhehcgjgoghdlhihaaaaaaaaaaaaaaablheaaafhdgdgphagfhdhcaafjgphcghcogbhagbgdgigfcohagjghcogcgbgdglgfgogecogigbgegpgphacogfhigfgdhfhegjgpgogfgoghgjgogfcohagihjhdgjgdgbgmemgbhjgfhccogfhihahcgfhdhdgjgpgoephagfhcgbhegphchdcofaepfahcgpgkgfgdheaaaaaaaaaaaaaaabacaaaffkaaakgphggfhcgmgpgbgegfgefkaabfhahcgpgdgfhdhdgjgoghecgbghepggfehfhagmgfhdfkaabehcgfhdhfgmhefdgjgoghgmgffehfhagmgfecgbghfkaaaehdhegbhcemaaahgdgpgmhfgngohdheaabfemgkgbhggbcphfhegjgmcpebhchcgbhjemgjhdhedlhihcaagc
|
|