Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Exception during parallel jobs execution using python


+
Jakub Glapa 2013-02-15, 16:27
+
Cheolsoo Park 2013-02-15, 22:05
Copy link to this message
-
Re: Exception during parallel jobs execution using python
Jakub Glapa 2013-03-29, 09:46
Hi Cheolsoo,
it's been a while but I finally filled a jira bug.
Unfortunately I'm not able to provide a reproducible use case.
I've described the problem I think in much detail.
Maybe somebody more experienced would know where to look for the root cause
of the problem or would recommend some debug techniques.

Thanks!

https://issues.apache.org/jira/browse/PIG-3263
--
regards,
Jakub Glapa
On Fri, Feb 15, 2013 at 10:05 PM, Cheolsoo Park <[EMAIL PROTECTED]>wrote:

> Hi Jakub,
>
> >> About 5 jobs fail like that out of 150.
>
> It sounds like a multi-threading bug in Pig to me. Do you mind filing a
> jira and attaching your script that reproduces the issue?
>
> Thanks,
> Cheolsoo
>
>
> On Fri, Feb 15, 2013 at 8:27 AM, Jakub Glapa <[EMAIL PROTECTED]>
> wrote:
>
> > I have a set of jobs to run with different parameters.
> >  I'm using Python to prepare the parameter sets and then I'm executing
> them
> > in batches with Pig.run(batchOfParams).
> >
> > The number of jobs is quite high but I'm running 4 of them at a time (4
> > sets of params in batchOfParams).
> >
> > From time to time I started noticing exception that terminate some of the
> > jobs.
> >
> > Can it be a problem with my UDF or is there a bug in in the Pig.run
> logic?
> >
> > The OrderQueryTokens is trivial. It just orders the words in string.
> >
> > About 5 jobs fail like that out of 150.
> >
> >
> > exception:
> >
> >
> > 2013-02-14 16:25:04,575 [main] ERROR
> org.apache.pig.scripting.BoundScript -
> > Pig pipeline failed to complete
> > java.util.concurrent.ExecutionException:
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
> > during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> > imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> > at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
> > at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
> > at
> >
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
> > at org.python.core.PyObject.__call__(PyObject.java:387)
> > at org.python.core.PyObject.__call__(PyObject.java:391)
> > at org.python.core.PyMethod.__call__(PyMethod.java:109)
> > at
> >
> >
> org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
> > at
> >
> >
> org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> > at org.python.core.PyTableCode.call(PyTableCode.java:165)
> > at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
> > at org.python.core.PyFunction.__call__(PyFunction.java:307)
> > at
> >
> >
> org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
> > at
> >
> >
> org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> > at org.python.core.PyTableCode.call(PyTableCode.java:165)
> > at org.python.core.PyCode.call(PyCode.java:18)
> > at org.python.core.Py.runCode(Py.java:1275)
> > at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
> > at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
> > at org.apache.pig.Main.runEmbeddedScript(Main.java:929)