Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Exception during parallel jobs execution using python


Copy link to this message
-
Re: Exception during parallel jobs execution using python
Hi Jakub,

>> About 5 jobs fail like that out of 150.

It sounds like a multi-threading bug in Pig to me. Do you mind filing a
jira and attaching your script that reproduces the issue?

Thanks,
Cheolsoo
On Fri, Feb 15, 2013 at 8:27 AM, Jakub Glapa <[EMAIL PROTECTED]> wrote:

> I have a set of jobs to run with different parameters.
>  I'm using Python to prepare the parameter sets and then I'm executing them
> in batches with Pig.run(batchOfParams).
>
> The number of jobs is quite high but I'm running 4 of them at a time (4
> sets of params in batchOfParams).
>
> From time to time I started noticing exception that terminate some of the
> jobs.
>
> Can it be a problem with my UDF or is there a bug in in the Pig.run logic?
>
> The OrderQueryTokens is trivial. It just orders the words in string.
>
> About 5 jobs fail like that out of 150.
>
>
> exception:
>
>
> 2013-02-14 16:25:04,575 [main] ERROR org.apache.pig.scripting.BoundScript -
> Pig pipeline failed to complete
> java.util.concurrent.ExecutionException:
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
> during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
> at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
> at
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
> at org.python.core.PyObject.__call__(PyObject.java:387)
> at org.python.core.PyObject.__call__(PyObject.java:391)
> at org.python.core.PyMethod.__call__(PyMethod.java:109)
> at
>
> org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
> at
>
> org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> at org.python.core.PyTableCode.call(PyTableCode.java:165)
> at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
> at org.python.core.PyFunction.__call__(PyFunction.java:307)
> at
>
> org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
> at
>
> org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> at org.python.core.PyTableCode.call(PyTableCode.java:165)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1275)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:929)
> at org.apache.pig.Main.run(Main.java:510)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000:
> Error during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]