Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> pig 0.11, Error while running pig script with Jython func which imports java class in cluster mode


Copy link to this message
-
pig 0.11, Error while running pig script with Jython func which imports java class in cluster mode
Hi, I'm testing my scripts in local mode then I run them in production
using oozie.
Locally everything works fine. My pig version is 0.11

When I run the same script in cluster mode, I do get exception on line
where jython udf is invoked. Here is my UDF, see it imports java class.
This class is IN runtime, I give 100%:

from ru.pig.geo import GSMCellCalculator

@outputSchema("centerLon:double, centerLat:double")
def calculateDropShapedCenter(lon, lat, maxDist, angleWidth, azimuth):
    print lon, lat, maxDist, angleWidth, azimuth
    spatialCoord = GSMCellCalculator.getDropShapeCenter(lon, lat,
float(maxDist), float(angleWidth), float(azimuth))
    return spatialCoord.longitude, spatialCoord.latitude

Here is a part of script:

register '$geoSpatialUdfs' using jython as udf;
/*
some code goes here....
*/
gsmCellProj = FOREACH gsmCellFixed GENERATE
                                         branchId,
                                         cellId, lac,
                                         lon, lat,
                                         (int)azimuth, (int)midDist,
(int)maxDist,
                                         cellType, (int)angWidth, gen,
startAng,
                                         angWidthFixed, startAngFixed,

 FLATTEN(udf.calculateDropShapedCenter(lon, lat, midDist, angWidth,
azimuth));

Here is the log STDOUT:

-07-10 17:33:30,729 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total
input paths to process : 1
2013-07-10 17:33:30,729 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total
input paths to process : 1
2013-07-10 17:33:30,737 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total
input paths (combined) to process : 1
2013-07-10 17:33:30,737 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total
input paths (combined) to process : 1
2013-07-10 17:33:31,562 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - HadoopJobId: job_201307101220_0154
2013-07-10 17:33:31,562 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - HadoopJobId: job_201307101220_0154
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Processing aliases
cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Processing aliases
cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - detailed locations: M:
gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - detailed locations: M:
gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - More information at:
http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
2013-07-10 17:33:31,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - More information at:
http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
Heart beat
2013-07-10 17:33:51,196 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 14% complete
2013-07-10 17:33:51,196 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 14% complete
2013-07-10 17:33:54,210 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 34% complete
2013-07-10 17:33:54,210 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 34% complete
2013-07-10 17:33:57,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 50% complete
2013-07-10 17:33:57,229 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 50% complete
2013-07-10 17:34:13,307 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:34:13,307 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
Heart beat
2013-07-10 17:34:31,899 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:34:31,899 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
Heart beat
2013-07-10 17:34:49,480 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:34:49,480 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:35:08,576 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
2013-07-10 17:35:08,576 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 83% complete
Heart beat
2013-07-10 17:35:26,170 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Ooops! Some job has failed! Specify -stop_on_failure if you want
Pig to stop immediately on failure.
2013-07-10 17:35:26,170 [main] WARN
org.apache.pig.backend.hadoo
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB