Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - pig12 job stuck in infinite loop


Copy link to this message
-
Re: pig12 job stuck in infinite loop
Suhas Satish 2014-03-12, 22:36
I was able to reproduce the issue on apache hadoop with latest apache pig
trunk, so I dont think its a MapR distribution -specific issue.

The infinite looping is through this section where only empty tuples () are
being generated with STATUS_OK, without a terminating condition being set.
I could not find a related JIRA, shall I open a new one for this?
protected void runPipeline(PhysicalOperator leaf) throws IOException,
InterruptedException {

     while(true){
            Result res = leaf.getNextTuple();
            log.debug(res.toString());
            log.debug("Suhas result's return status="+ res.returnStatus);
           * if(res.returnStatus==POStatus.STATUS_OK){*
*                collect(outputCollector,(Tuple)res.result);*
*                continue;*
*            }*

            if(res.returnStatus==POStatus.STATUS_EOP) {
                return;
            }

            if(res.returnStatus==POStatus.STATUS_NULL)
                continue;

            if(res.returnStatus==POStatus.STATUS_ERR){
                // remember that we had an issue so that in
                // close() we can do the right thing
                errorInMap  = true;
                // if there is an errmessage use it
                String errMsg;
                if(res.result != null) {
                    errMsg = "Received Error while " +
                    "processing the map plan: " + res.result;
                } else {
                    errMsg = "Received Error while " +
                    "processing the map plan.";
                }

                int errCode = 2055;
                ExecException ee = new ExecException(errMsg, errCode,
PigException.BUG);
                throw ee;
            }
        }

    }
=================================================================
How to interpret this physical plan from the leaf?

2014-03-12 11:32:36,609 DEBUG
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map:
*gpWeekRanked:
New For Each()[bag] *- scope-29
|
|---New For Each(false,true)[tuple] - scope-28
    |   |
    |   Project[long][0] - scope-26
    |   |
    |   Project[bag][2] - scope-27
    |
    |---pWeekRanked: PORank[tuple] - scope-25
        |   |
        |   Project[int][0] - scope-22
2014-03-12 11:32:36,610 INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map:
Aliases being processed per job phase*(AliasName[line,offset]):* *M:
gpWeekRanked[11,15] *C:  R:
=================================================================
pig script -

tWeek = LOAD '/tmp/test_data.txt' USING PigStorage ('|') AS (WEEK:int,
DESCRIPTION:chararray, END_DATE:chararray, PERIOD:int);

gTWeek = FOREACH tWeek GENERATE WEEK AS WEEK, PERIOD AS PERIOD;

pWeek = FILTER gTWeek BY (PERIOD == 201312);

pWeekRanked = RANK pWeek BY WEEK ASC DENSE;

gpWeekRanked = FOREACH pWeekRanked GENERATE $0;

store gpWeekRanked INTO 'gpWeekRanked2';

describe gpWeekRanked2;
=================================================================

Cheers,
Suhas.
On Fri, Mar 7, 2014 at 10:20 AM, Cheolsoo Park <[EMAIL PROTECTED]> wrote:
 20100101|1 WEEK ENDING 01/02/10|2010-Jan-02|201001|1468
20100102|1 WEEK ENDING 01/09/10|2010-Jan-09|201001|1469
20100103|1 WEEK ENDING 01/16/10|2010-Jan-16|201001|1470
20100104|1 WEEK ENDING 01/23/10|2010-Jan-23|201001|1471
20100205|1 WEEK ENDING 01/30/10|2010-Jan-30|201002|1472
20100206|1 WEEK ENDING 02/06/10|2010-Feb-06|201002|1473
20100207|1 WEEK ENDING 02/13/10|2010-Feb-13|201002|1474
20100208|1 WEEK ENDING 02/20/10|2010-Feb-20|201002|1475
20100309|1 WEEK ENDING 02/27/10|2010-Feb-27|201003|1476
20100310|1 WEEK ENDING 03/06/10|2010-Mar-06|201003|1477
20100311|1 WEEK ENDING 03/13/10|2010-Mar-13|201003|1478
20100312|1 WEEK ENDING 03/20/10|2010-Mar-20|201003|1479
20100413|1 WEEK ENDING 03/27/10|2010-Mar-27|201004|1480
20100414|1 WEEK ENDING 04/03/10|2010-Apr-03|201004|1481
20100415|1 WEEK ENDING 04/10/10|2010-Apr-10|201004|1482
20100416|1 WEEK ENDING 04/17/10|2010-Apr-17|201004|1483
20100517|1 WEEK ENDING 04/24/10|2010-Apr-24|201005|1484
20100518|1 WEEK ENDING 05/01/10|2010-May-01|201005|1485
20100519|1 WEEK ENDING 05/08/10|2010-May-08|201005|1486
20100520|1 WEEK ENDING 05/15/10|2010-May-15|201005|1487
20100621|1 WEEK ENDING 05/22/10|2010-May-22|201006|1488
20100622|1 WEEK ENDING 05/29/10|2010-May-29|201006|1489
20100623|1 WEEK ENDING 06/05/10|2010-Jun-05|201006|1490
20100624|1 WEEK ENDING 06/12/10|2010-Jun-12|201006|1491
20100725|1 WEEK ENDING 06/19/10|2010-Jun-19|201007|1492
20100726|1 WEEK ENDING 06/26/10|2010-Jun-26|201007|1493
20100727|1 WEEK ENDING 07/03/10|2010-Jul-03|201007|1494
20100728|1 WEEK ENDING 07/10/10|2010-Jul-10|201007|1495
20100829|1 WEEK ENDING 07/17/10|2010-Jul-17|201008|1496
20100830|1 WEEK ENDING 07/24/10|2010-Jul-24|201008|1497
20100831|1 WEEK ENDING 07/31/10|2010-Jul-31|201008|1498
20100832|1 WEEK ENDING 08/07/10|2010-Aug-07|201008|1499
20100933|1 WEEK ENDING 08/14/10|2010-Aug-14|201009|1500
20100934|1 WEEK ENDING 08/21/10|2010-Aug-21|201009|1501
20100935|1 WEEK ENDING 08/28/10|2010-Aug-28|201009|1502
20100936|1 WEEK ENDING 09/04/10|2010-Sep-04|201009|1503
20101037|1 WEEK ENDING 09/11/10|2010-Sep-11|201010|1504
20101038|1 WEEK ENDING 09/18/10|2010-Sep-18|201010|1505
20101039|1 WEEK ENDING 09/25/10|2010-Sep-25|201010|1506
20101040|1 WEEK ENDING 10/02/10|2010-Oct-02|201010|1507
20101141|1 WEEK ENDING 10/09/10|2010-Oct-09|201011|1508
20101142|1 WEEK ENDING 10/16/10|2010-Oct-16|201011|1509
20101143|1 WEEK ENDING 10/23/10|2010-Oct-23|201011|1510
20101144|1 WEEK ENDING 10/30/10|2010-Oct-30|201011|1511
20101245|1 WEEK ENDING 11/06/10|2010-Nov-06|201012|1512
20101246|1 WEEK ENDING 11/13/10|2010-Nov-13|201012|1513
20101247|1 WEEK ENDING 11/20/10|2010-Nov-20|201012|1514
20101248|1 WEEK ENDING 11/27/10|2010-Nov-27|201012|1515
20101349|1 WEEK ENDING 12/04/10|2010-Dec-04|201013|1516
20101350|1 WEEK ENDING 12/11/10|2010-Dec-11|201013|1517
20101351|1 WEEK ENDING 12/18/10|2010-Dec-18|201013|1518
20101352|1 WEEK ENDING 12/25/10|2010