Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Exception during parallel jobs execution using python


Copy link to this message
-
Re: Exception during parallel jobs execution using python
Hi Jakub,

>> About 5 jobs fail like that out of 150.

It sounds like a multi-threading bug in Pig to me. Do you mind filing a
jira and attaching your script that reproduces the issue?

Thanks,
Cheolsoo
On Fri, Feb 15, 2013 at 8:27 AM, Jakub Glapa <[EMAIL PROTECTED]> wrote:

> I have a set of jobs to run with different parameters.
>  I'm using Python to prepare the parameter sets and then I'm executing them
> in batches with Pig.run(batchOfParams).
>
> The number of jobs is quite high but I'm running 4 of them at a time (4
> sets of params in batchOfParams).
>
> From time to time I started noticing exception that terminate some of the
> jobs.
>
> Can it be a problem with my UDF or is there a bug in in the Pig.run logic?
>
> The OrderQueryTokens is trivial. It just orders the words in string.
>
> About 5 jobs fail like that out of 150.
>
>
> exception:
>
>
> 2013-02-14 16:25:04,575 [main] ERROR org.apache.pig.scripting.BoundScript -
> Pig pipeline failed to complete
> java.util.concurrent.ExecutionException:
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
> during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
> at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
> at
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
> at org.python.core.PyObject.__call__(PyObject.java:387)
> at org.python.core.PyObject.__call__(PyObject.java:391)
> at org.python.core.PyMethod.__call__(PyMethod.java:109)
> at
>
> org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
> at
>
> org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> at org.python.core.PyTableCode.call(PyTableCode.java:165)
> at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
> at org.python.core.PyFunction.__call__(PyFunction.java:307)
> at
>
> org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
> at
>
> org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> at org.python.core.PyTableCode.call(PyTableCode.java:165)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1275)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
> at
>
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:929)
> at org.apache.pig.Main.run(Main.java:510)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000:
> Error during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB