Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Exception during parallel jobs execution using python


+
Jakub Glapa 2013-02-15, 16:27
+
Cheolsoo Park 2013-02-15, 22:05
Copy link to this message
-
Re: Exception during parallel jobs execution using python
Hi Cheolsoo,
it's been a while but I finally filled a jira bug.
Unfortunately I'm not able to provide a reproducible use case.
I've described the problem I think in much detail.
Maybe somebody more experienced would know where to look for the root cause
of the problem or would recommend some debug techniques.

Thanks!

https://issues.apache.org/jira/browse/PIG-3263
--
regards,
Jakub Glapa
On Fri, Feb 15, 2013 at 10:05 PM, Cheolsoo Park <[EMAIL PROTECTED]>wrote:

> Hi Jakub,
>
> >> About 5 jobs fail like that out of 150.
>
> It sounds like a multi-threading bug in Pig to me. Do you mind filing a
> jira and attaching your script that reproduces the issue?
>
> Thanks,
> Cheolsoo
>
>
> On Fri, Feb 15, 2013 at 8:27 AM, Jakub Glapa <[EMAIL PROTECTED]>
> wrote:
>
> > I have a set of jobs to run with different parameters.
> >  I'm using Python to prepare the parameter sets and then I'm executing
> them
> > in batches with Pig.run(batchOfParams).
> >
> > The number of jobs is quite high but I'm running 4 of them at a time (4
> > sets of params in batchOfParams).
> >
> > From time to time I started noticing exception that terminate some of the
> > jobs.
> >
> > Can it be a problem with my UDF or is there a bug in in the Pig.run
> logic?
> >
> > The OrderQueryTokens is trivial. It just orders the words in string.
> >
> > About 5 jobs fail like that out of 150.
> >
> >
> > exception:
> >
> >
> > 2013-02-14 16:25:04,575 [main] ERROR
> org.apache.pig.scripting.BoundScript -
> > Pig pipeline failed to complete
> > java.util.concurrent.ExecutionException:
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
> > during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> > imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> > at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
> > at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
> > at
> >
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
> > at org.python.core.PyObject.__call__(PyObject.java:387)
> > at org.python.core.PyObject.__call__(PyObject.java:391)
> > at org.python.core.PyMethod.__call__(PyMethod.java:109)
> > at
> >
> >
> org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
> > at
> >
> >
> org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> > at org.python.core.PyTableCode.call(PyTableCode.java:165)
> > at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
> > at org.python.core.PyFunction.__call__(PyFunction.java:307)
> > at
> >
> >
> org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
> > at
> >
> >
> org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> > at org.python.core.PyTableCode.call(PyTableCode.java:165)
> > at org.python.core.PyCode.call(PyCode.java:18)
> > at org.python.core.Py.runCode(Py.java:1275)
> > at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
> > at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
> > at org.apache.pig.Main.runEmbeddedScript(Main.java:929)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB