Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> pig12 job stuck in infinite loop


Copy link to this message
-
Re: pig12 job stuck in infinite loop
I was able to reproduce the issue on apache hadoop with latest apache pig
trunk, so I dont think its a MapR distribution -specific issue.

The infinite looping is through this section where only empty tuples () are
being generated with STATUS_OK, without a terminating condition being set.
I could not find a related JIRA, shall I open a new one for this?
protected void runPipeline(PhysicalOperator leaf) throws IOException,
InterruptedException {

     while(true){
            Result res = leaf.getNextTuple();
            log.debug(res.toString());
            log.debug("Suhas result's return status="+ res.returnStatus);
           * if(res.returnStatus==POStatus.STATUS_OK){*
*                collect(outputCollector,(Tuple)res.result);*
*                continue;*
*            }*

            if(res.returnStatus==POStatus.STATUS_EOP) {
                return;
            }

            if(res.returnStatus==POStatus.STATUS_NULL)
                continue;

            if(res.returnStatus==POStatus.STATUS_ERR){
                // remember that we had an issue so that in
                // close() we can do the right thing
                errorInMap  = true;
                // if there is an errmessage use it
                String errMsg;
                if(res.result != null) {
                    errMsg = "Received Error while " +
                    "processing the map plan: " + res.result;
                } else {
                    errMsg = "Received Error while " +
                    "processing the map plan.";
                }

                int errCode = 2055;
                ExecException ee = new ExecException(errMsg, errCode,
PigException.BUG);
                throw ee;
            }
        }

    }
=================================================================
How to interpret this physical plan from the leaf?

2014-03-12 11:32:36,609 DEBUG
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map:
*gpWeekRanked:
New For Each()[bag] *- scope-29
|
|---New For Each(false,true)[tuple] - scope-28
    |   |
    |   Project[long][0] - scope-26
    |   |
    |   Project[bag][2] - scope-27
    |
    |---pWeekRanked: PORank[tuple] - scope-25
        |   |
        |   Project[int][0] - scope-22
2014-03-12 11:32:36,610 INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map:
Aliases being processed per job phase*(AliasName[line,offset]):* *M:
gpWeekRanked[11,15] *C:  R:
=================================================================
pig script -

tWeek = LOAD '/tmp/test_data.txt' USING PigStorage ('|') AS (WEEK:int,
DESCRIPTION:chararray, END_DATE:chararray, PERIOD:int);

gTWeek = FOREACH tWeek GENERATE WEEK AS WEEK, PERIOD AS PERIOD;

pWeek = FILTER gTWeek BY (PERIOD == 201312);

pWeekRanked = RANK pWeek BY WEEK ASC DENSE;

gpWeekRanked = FOREACH pWeekRanked GENERATE $0;

store gpWeekRanked INTO 'gpWeekRanked2';

describe gpWeekRanked2;
=================================================================

Cheers,
Suhas.
On Fri, Mar 7, 2014 at 10:20 AM, Cheolsoo Park <[EMAIL PROTECTED]> wrote:
 20100101|1 WEEK ENDING 01/02/10|2010-Jan-02|201001|1468
20100102|1 WEEK ENDING 01/09/10|2010-Jan-09|201001|1469
20100103|1 WEEK ENDING 01/16/10|2010-Jan-16|201001|1470
20100104|1 WEEK ENDING 01/23/10|2010-Jan-23|201001|1471
20100205|1 WEEK ENDING 01/30/10|2010-Jan-30|201002|1472
20100206|1 WEEK ENDING 02/06/10|2010-Feb-06|201002|1473
20100207|1 WEEK ENDING 02/13/10|2010-Feb-13|201002|1474
20100208|1 WEEK ENDING 02/20/10|2010-Feb-20|201002|1475
20100309|1 WEEK ENDING 02/27/10|2010-Feb-27|201003|1476
20100310|1 WEEK ENDING 03/06/10|2010-Mar-06|201003|1477
20100311|1 WEEK ENDING 03/13/10|2010-Mar-13|201003|1478
20100312|1 WEEK ENDING 03/20/10|2010-Mar-20|201003|1479
20100413|1 WEEK ENDING 03/27/10|2010-Mar-27|201004|1480
20100414|1 WEEK ENDING 04/03/10|2010-Apr-03|201004|1481
20100415|1 WEEK ENDING 04/10/10|2010-Apr-10|201004|1482
20100416|1 WEEK ENDING 04/17/10|2010-Apr-17|201004|1483
20100517|1 WEEK ENDING 04/24/10|2010-Apr-24|201005|1484
20100518|1 WEEK ENDING 05/01/10|2010-May-01|201005|1485
20100519|1 WEEK ENDING 05/08/10|2010-May-08|201005|1486
20100520|1 WEEK ENDING 05/15/10|2010-May-15|201005|1487
20100621|1 WEEK ENDING 05/22/10|2010-May-22|201006|1488
20100622|1 WEEK ENDING 05/29/10|2010-May-29|201006|1489
20100623|1 WEEK ENDING 06/05/10|2010-Jun-05|201006|1490
20100624|1 WEEK ENDING 06/12/10|2010-Jun-12|201006|1491
20100725|1 WEEK ENDING 06/19/10|2010-Jun-19|201007|1492
20100726|1 WEEK ENDING 06/26/10|2010-Jun-26|201007|1493
20100727|1 WEEK ENDING 07/03/10|2010-Jul-03|201007|1494
20100728|1 WEEK ENDING 07/10/10|2010-Jul-10|201007|1495
20100829|1 WEEK ENDING 07/17/10|2010-Jul-17|201008|1496
20100830|1 WEEK ENDING 07/24/10|2010-Jul-24|201008|1497
20100831|1 WEEK ENDING 07/31/10|2010-Jul-31|201008|1498
20100832|1 WEEK ENDING 08/07/10|2010-Aug-07|201008|1499
20100933|1 WEEK ENDING 08/14/10|2010-Aug-14|201009|1500
20100934|1 WEEK ENDING 08/21/10|2010-Aug-21|201009|1501
20100935|1 WEEK ENDING 08/28/10|2010-Aug-28|201009|1502
20100936|1 WEEK ENDING 09/04/10|2010-Sep-04|201009|1503
20101037|1 WEEK ENDING 09/11/10|2010-Sep-11|201010|1504
20101038|1 WEEK ENDING 09/18/10|2010-Sep-18|201010|1505
20101039|1 WEEK ENDING 09/25/10|2010-Sep-25|201010|1506
20101040|1 WEEK ENDING 10/02/10|2010-Oct-02|201010|1507
20101141|1 WEEK ENDING 10/09/10|2010-Oct-09|201011|1508
20101142|1 WEEK ENDING 10/16/10|2010-Oct-16|201011|1509
20101143|1 WEEK ENDING 10/23/10|2010-Oct-23|201011|1510
20101144|1 WEEK ENDING 10/30/10|2010-Oct-30|201011|1511
20101245|1 WEEK ENDING 11/06/10|2010-Nov-06|201012|1512
20101246|1 WEEK ENDING 11/13/10|2010-Nov-13|201012|1513
20101247|1 WEEK ENDING 11/20/10|2010-Nov-20|201012|1514
20101248|1 WEEK ENDING 11/27/10|2010-Nov-27|201012|1515
20101349|1 WEEK ENDING 12/04/10|2010-Dec-04|201013|1516
20101350|1 WEEK ENDING 12/11/10|2010-Dec-11|201013|1517
20101351|1 WEEK ENDING 12/18/10|2010-Dec-18|201013|1518
20101352|1 WEEK ENDING 12/25/10|2010
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB