Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> pig 0.11, Error while running pig script with Jython func which imports java class in cluster mode


Copy link to this message
-
pig 0.11, Error while running pig script with Jython func which imports java class in cluster mode
Hi, I'm testing my scripts in local mode then I run them in production
using oozie.
Locally everything works fine. My pig version is 0.11

When I run the same script in cluster mode, I do get exception on line
where jython udf is invoked. Here is my UDF, see it imports java class.
This class is IN runtime, I give 100%:

from ru.pig.geo import GSMCellCalculator

@outputSchema("centerLon:double, centerLat:double")
def calculateDropShapedCenter(lon, lat, maxDist, angleWidth, azimuth):
    print lon, lat, maxDist, angleWidth, azimuth
    spatialCoord = GSMCellCalculator.getDropShapeCenter(lon, lat,
float(maxDist), float(angleWidth), float(azimuth))
    return spatialCoord.longitude, spatialCoord.latitude

Here is a part of script:

register '$geoSpatialUdfs' using jython as udf;
/*
some code goes here....
*/
gsmCellProj = FOREACH gsmCellFixed GENERATE
                                         branchId,
                                         cellId, lac,
                                         lon, lat,
                                         (int)azimuth, (int)midDist,
(int)maxDist,
                                         cellType, (int)angWidth, gen,
startAng,
                                         angWidthFixed, startAngFixed,

 FLATTEN(udf.calculateDropShapedCenter(lon, lat, midDist, angWidth,
azimuth));

Here is the log STDOUT:

-07-10 17:33:30,729 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total
input paths to process : 1
2013-07-10 17:33:30,729 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total
input paths to process : 1
2013-07-10 17:33:30,737 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total
input paths (combined) to process : 1
2013-07-10 17:33:30,737 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total
input paths (combined) to process : 1
2013-07-10 17:33:31,562 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - HadoopJobId: job_201307101220_0154
2013-07-10 17:33:31,562 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - HadoopJobId: job_201307101220_0154
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Processing aliases
cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Processing aliases
cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - detailed locations: M:
gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - detailed locations: M:
gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - More information at:
http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - More information at:
http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
Heart beat
2013-07-10 17:33:51,196 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 14% complete
2013-07-10 17:33:51,196 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 14% complete
2013-07-10 17:33:54,210 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 34% complete
2013-07-10 17:33:54,210 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 34% complete
2013-07-10 17:33:57,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 50% complete
2013-07-10 17:33:57,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 50% complete
2013-07-10 17:34:13,307 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:34:13,307 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
Heart beat
2013-07-10 17:34:31,899 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:34:31,899 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
Heart beat
2013-07-10 17:34:49,480 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:34:49,480 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:35:08,576 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:35:08,576 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
Heart beat
2013-07-10 17:35:26,170 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Ooops! Some job has failed! Specify -stop_on_failure if you want
Pig to stop immediately on failure.
2013-07-10 17:35:26,170 [main] WARN
org.apache.pig.backend.hadoo