Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Exception during parallel jobs execution using python


Copy link to this message
-
Exception during parallel jobs execution using python
Jakub Glapa 2013-02-15, 16:27
I have a set of jobs to run with different parameters.
 I'm using Python to prepare the parameter sets and then I'm executing them
in batches with Pig.run(batchOfParams).

The number of jobs is quite high but I'm running 4 of them at a time (4
sets of params in batchOfParams).

>From time to time I started noticing exception that terminate some of the
jobs.

Can it be a problem with my UDF or is there a bug in in the Pig.run logic?

The OrderQueryTokens is trivial. It just orders the words in string.

About 5 jobs fail like that out of 150.
exception:
2013-02-14 16:25:04,575 [main] ERROR org.apache.pig.scripting.BoundScript -
Pig pipeline failed to complete
java.util.concurrent.ExecutionException:
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
at
org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
at org.python.core.PyObject.__call__(PyObject.java:387)
at org.python.core.PyObject.__call__(PyObject.java:391)
at org.python.core.PyMethod.__call__(PyMethod.java:109)
at
org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
at
org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
at org.python.core.PyFunction.__call__(PyFunction.java:307)
at
org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
at
org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1275)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
at
org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
at
org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
at
org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
at org.apache.pig.Main.runEmbeddedScript(Main.java:929)
at org.apache.pig.Main.run(Main.java:510)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000:
Error during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1606)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1549)
at org.apache.pig.PigServer.registerQuery(PigServer.java:549)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:971)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:190)
at
org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:342)
at
org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:318)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: Failed to parse: Pig script failed to parse:
<line 36, column 76> Failed to generate logical plan. Nested exception:
java.lang.RuntimeException: Cannot instantiate: my.pig.udf.OrderQueryTokens
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1598)
... 12 more
Caused by:
<line 36, column 76> Failed to generate logical plan. Nested exception:
java.lang.RuntimeException: Cannot instantiate: my.pig.udf.OrderQueryTokens
at
org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:980)
at
org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:7316)
at
org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:8857)
at
org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:8632)
at
org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:7984)
at
org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:5962)
at
org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:14101)
at
org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:12493)
at
org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:12360)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1577)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(Logi
+
Cheolsoo Park 2013-02-15, 22:05
+
Jakub Glapa 2013-03-29, 09:46