Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> about pig-0.8.1 unit test


Copy link to this message
-
Re:Re: about pig-0.8.1 unit test
Hello,
For test case TestPruneColumn:
    public void testMapKeyInSplit1() throws Exception {
        pigServer.registerQuery("A = load '"+ Util.generateURI(tmpFile12.toString(), pigServer.getPigContext()) + "' as (m:map[]);");
        pigServer.registerQuery("B = foreach A generate m#'key1' as key1;");
        pigServer.registerQuery("C = foreach A generate m#'key2' as key2;");
        pigServer.registerQuery("D = join B by key1, C by key2;");

        Iterator<Tuple> iter = pigServer.openIterator("D");

        assertTrue(iter.hasNext());
        Tuple t = iter.next();
        assertTrue(t.size()==2);
        assertTrue(t.get(0).toString().equals("2"));
        assertTrue(t.get(1).toString().equals("2"));

        assertFalse(iter.hasNext());

        assertTrue(checkLogFileMessage(new String[]{"Map key required for A: $0->[key2, key1]"}));
    }
I have some questions:
1. What content will be stored in logFile in methond checkLogFileMessage?
2. With other JDK but not SUN JDK, I found the variable item contain string like "Map key required for A: $0->[key1, key2]". I think this is caused by pigServer.registerQuery method which invoke method parse. And parse method is based on HashMap structure.Could you please tell me what HashMap methods will be invoked and cause the different output LogicalPlan in parse method?
    public LogicalPlan parse(String scope,
                             String query,
                             Map<LogicalOperator, LogicalPlan> aliases,
                             Map<OperatorKey, LogicalOperator> opTable,
                             Map<String, LogicalOperator> aliasOp,
                             int start,
                             Map<String, String> fileNameMap)
        throws IOException, ParseException {
        ByteArrayInputStream in = new ByteArrayInputStream(query.getBytes());      
        QueryParser parser = new QueryParser(in, pigContext, scope, aliases, opTable,
                                             aliasOp, start, fileNameMap);
        return parser.Parse();      
    }
3.The same situation also happened in:
Testcase: testMapKey2 took 8.133 sec
Should be:"Columns pruned for A: $0",
                "Map key required for A: $1->[key2, key1]"
But is:"Columns pruned for A: $0"
          "Map key required for A: $1->[key1, key2]"

Testcase: testMapKey3 took 8.815 sec
Should be:{([key2#2,key1#1],1),([key2#4,key1#2],2)}
But is:{([key1#1,key2#2],1),([key1#2,key2#4],2)}
Testcase: testSharedSchemaObject took 9.15 sec
Should be:([2#1,1#1])
But is:([1#1,2#1])
Are these results also correct?
At 2011-08-19 01:50:20,"Thejas Nair" <[EMAIL PROTECTED]> wrote:
>yes, even in this case, both outputs look correct.
>-Thejas
>
>
>
>On 8/18/11 6:05 AM, lulynn_2008 wrote:
>> Thejas  Nair,
>>
>> Thank you
>> Actually there are other unit tests failed because of different order
>> output. During parse the pig command, a HashMap variable "aliases" will
>> be used in
>> src-gen/org/apache/pig/impl/logicalLayer/parser/QueryParser.java. I
>> think this unit test failure is also caused by HashMap problem. What is
>> your opinion?
>> Here are two failure unit test examples:
>> TestLogToPhyCompiler: Query: split (load 'a') into x if $0 < '7', y if
>> $0 > '7';
>> Testcase: testSplit took 0.198 sec
>> FAILED
>> IBM JDK:
>> x: Filter[tuple] - Test-Plan-Builder-240
>> | |
>> | Less Than[boolean] - Test-Plan-Builder-243
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-241
>> | |
>> | |---Constant(7) - Test-Plan-Builder-242
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>>
>> y: Filter[tuple] - Test-Plan-Builder-244
>> | |
>> | Greater Than[boolean] - Test-Plan-Builder-247
>> | |
>> | |---Project[bytearray][0] - Test-Plan-Builder-245
>> | |
>> | |---Constant(7) - Test-Plan-Builder-246
>> |
>> |---Split - Test-Plan-Builder-239
>> |
>> |---229: Load()
>> SUN JDK:
>> y: Filter[tuple] - Test-Plan-Builder-240
>> | |
>> | Greater Than[boolean] - Test-Plan-Builder-243
>> | |
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB