I have a set of jobs to run with different parameters.
 I'm using Python to prepare the parameter sets and then I'm executing them
in batches with Pig.run(batchOfParams).

The number of jobs is quite high but I'm running 4 of them at a time (4
sets of params in batchOfParams).

>From time to time I started noticing exception that terminate some of the
jobs.

Can it be a problem with my UDF or is there a bug in in the Pig.run logic?

The OrderQueryTokens is trivial. It just orders the words in string.

About 5 jobs fail like that out of 150.
exception:
2013-02-14 16:25:04,575 [main] ERROR org.apache.pig.scripting.BoundScript -
Pig pipeline failed to complete
java.util.concurrent.ExecutionException:
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
at
org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
at org.python.core.PyObject.__call__(PyObject.java:387)
at org.python.core.PyObject.__call__(PyObject.java:391)
at org.python.core.PyMethod.__call__(PyMethod.java:109)
at
org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
at
org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
at org.python.core.PyFunction.__call__(PyFunction.java:307)
at
org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
at
org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1275)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
at
org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
at
org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
at
org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
at org.apache.pig.Main.runEmbeddedScript(Main.java:929)
at org.apache.pig.Main.run(Main.java:510)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000:
Error during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1606)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1549)
at org.apache.pig.PigServer.registerQuery(PigServer.java:549)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:971)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:190)
at
org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:342)
at
org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:318)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: Failed to parse: Pig script failed to parse:
<line 36, column 76> Failed to generate logical plan. Nested exception:
java.lang.RuntimeException: Cannot instantiate: my.pig.udf.OrderQueryTokens
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1598)
... 12 more
Caused by:
<line 36, column 76> Failed to generate logical plan. Nested exception:
java.lang.RuntimeException: Cannot instantiate: my.pig.udf.OrderQueryTokens
at
org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:980)
at
org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:7316)
at
org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:8857)
at
org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:8632)
at
org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:7984)
at
org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:5962)
at
org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:14101)
at
org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:12493)
at
org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:12360)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1577)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(Logi

Hi Jakub,

>> About 5 jobs fail like that out of 150.

It sounds like a multi-threading bug in Pig to me. Do you mind filing a
jira and attaching your script that reproduces the issue?

Thanks,
Cheolsoo
On Fri, Feb 15, 2013 at 8:27 AM, Jakub Glapa <[EMAIL PROTECTED]> wrote:

> I have a set of jobs to run with different parameters.
>  I'm using Python to prepare the parameter sets and then I'm executing them
> in batches with Pig.run(batchOfParams).
>
> The number of jobs is quite high but I'm running 4 of them at a time (4
> sets of params in batchOfParams).
>
> From time to time I started noticing exception that terminate some of the
> jobs.
>
> Can it be a problem with my UDF or is there a bug in in the Pig.run logic?
>
> The OrderQueryTokens is trivial. It just orders the words in string.
>
> About 5 jobs fail like that out of 150.
>
>
> exception:
>
>
> 2013-02-14 16:25:04,575 [main] ERROR org.apache.pig.scripting.BoundScript -
> Pig pipeline failed to complete
> java.util.concurrent.ExecutionException:
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
> during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
> at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
> at
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
> at org.python.core.PyObject.__call__(PyObject.java:387)
> at org.python.core.PyObject.__call__(PyObject.java:391)
> at org.python.core.PyMethod.__call__(PyMethod.java:109)
> at
>
> org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
> at
>
> org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> at org.python.core.PyTableCode.call(PyTableCode.java:165)
> at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
> at org.python.core.PyFunction.__call__(PyFunction.java:307)
> at
>
> org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
> at
>
> org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> at org.python.core.PyTableCode.call(PyTableCode.java:165)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1275)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:929)
> at org.apache.pig.Main.run(Main.java:510)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000:
> Error during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]

Hi Cheolsoo,
it's been a while but I finally filled a jira bug.
Unfortunately I'm not able to provide a reproducible use case.
I've described the problem I think in much detail.
Maybe somebody more experienced would know where to look for the root cause
of the problem or would recommend some debug techniques.

Thanks!

https://issues.apache.org/jira/browse/PIG-3263
--
regards,
Jakub Glapa
On Fri, Feb 15, 2013 at 10:05 PM, Cheolsoo Park <[EMAIL PROTECTED]>wrote:

> Hi Jakub,
>
> >> About 5 jobs fail like that out of 150.
>
> It sounds like a multi-threading bug in Pig to me. Do you mind filing a
> jira and attaching your script that reproduces the issue?
>
> Thanks,
> Cheolsoo
>
>
> On Fri, Feb 15, 2013 at 8:27 AM, Jakub Glapa <[EMAIL PROTECTED]>
> wrote:
>
> > I have a set of jobs to run with different parameters.
> >  I'm using Python to prepare the parameter sets and then I'm executing
> them
> > in batches with Pig.run(batchOfParams).
> >
> > The number of jobs is quite high but I'm running 4 of them at a time (4
> > sets of params in batchOfParams).
> >
> > From time to time I started noticing exception that terminate some of the
> > jobs.
> >
> > Can it be a problem with my UDF or is there a bug in in the Pig.run
> logic?
> >
> > The OrderQueryTokens is trivial. It just orders the words in string.
> >
> > About 5 jobs fail like that out of 150.
> >
> >
> > exception:
> >
> >
> > 2013-02-14 16:25:04,575 [main] ERROR
> org.apache.pig.scripting.BoundScript -
> > Pig pipeline failed to complete
> > java.util.concurrent.ExecutionException:
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
> > during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> > imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> > at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
> > at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
> > at
> >
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
> > at org.python.core.PyObject.__call__(PyObject.java:387)
> > at org.python.core.PyObject.__call__(PyObject.java:391)
> > at org.python.core.PyMethod.__call__(PyMethod.java:109)
> > at
> >
> >
> org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
> > at
> >
> >
> org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> > at org.python.core.PyTableCode.call(PyTableCode.java:165)
> > at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
> > at org.python.core.PyFunction.__call__(PyFunction.java:307)
> > at
> >
> >
> org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
> > at
> >
> >
> org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> > at org.python.core.PyTableCode.call(PyTableCode.java:165)
> > at org.python.core.PyCode.call(PyCode.java:18)
> > at org.python.core.Py.runCode(Py.java:1275)
> > at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
> > at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
> > at org.apache.pig.Main.runEmbeddedScript(Main.java:929)