Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 13781: Changes to add support for streaming_python udfs.


Copy link to this message
-
Re: Review Request 13781: Changes to add support for streaming_python udfs.
Daniel Dai 2013-09-16, 04:18


> On Sept. 7, 2013, 12:04 a.m., Daniel Dai wrote:
> > src/org/apache/pig/builtin/StreamingUDF.java, line 169
> > <https://reviews.apache.org/r/13781/diff/2/?file=348276#file348276line169>
> >
> >     Are those files only for illustrate? Why not just use a random tmp file? Why do we distinguish exectype in illustrate? It only run locally.
>
> Jeremy Karn wrote:
>     I renamed the class to remove Illustrate from the name.  This is actually used both for illustrate (where we capture the output only from the last run) and in real runs (where we write the output to the user logs).  By putting it in the user logs you can get the logs from the task tracker web interface.  We could use temp files but I like the more human readable names.

Does the log file location hack also work in Hadoop2?
> On Sept. 7, 2013, 12:04 a.m., Daniel Dai wrote:
> > test/unit-tests, line 73
> > <https://reviews.apache.org/r/13781/diff/2/?file=348302#file348302line73>
> >
> >     This is not true unit test, should remove from the list
>
> Jeremy Karn wrote:
>     Can it go in commit-tests?

The goal for commit-tests to complete within 10 min. So we don't tend to include end-to-end unit test cases.
- Daniel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13781/#review25964
-----------------------------------------------------------
On Sept. 7, 2013, 3:39 p.m., Jeremy Karn wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13781/
> -----------------------------------------------------------
>
> (Updated Sept. 7, 2013, 3:39 p.m.)
>
>
> Review request for pig.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Changes for PIG-2417 (https://issues.apache.org/jira/browse/PIG-2417)
>
>
> Diffs
> -----
>
>   build.xml 7e22192
>   src/org/apache/pig/impl/builtin/StreamingUDF.java PRE-CREATION
>   src/org/apache/pig/impl/streaming/ExecutableManager.java cf79c83
>   src/org/apache/pig/impl/streaming/OutputHandler.java 6e9262a
>   src/org/apache/pig/impl/streaming/PigToStreamUDF.java PRE-CREATION
>   src/org/apache/pig/impl/streaming/StreamUDFToPig.java PRE-CREATION
>   src/org/apache/pig/impl/streaming/StreamingDelimiters.java PRE-CREATION
>   src/org/apache/pig/impl/streaming/StreamingUDFException.java PRE-CREATION
>   src/org/apache/pig/impl/streaming/StreamingUDFInputHandler.java PRE-CREATION
>   src/org/apache/pig/impl/streaming/StreamingUDFOutputHandler.java PRE-CREATION
>   src/org/apache/pig/impl/streaming/StreamingUDFOutputSchemaException.java PRE-CREATION
>   src/org/apache/pig/impl/streaming/StreamingUtil.java PRE-CREATION
>   src/org/apache/pig/impl/util/JarManager.java 5c4acb0
>   src/org/apache/pig/impl/util/StorageUtil.java dcb62ec
>   src/org/apache/pig/scripting/ScriptEngine.java 29a9e1f
>   src/org/apache/pig/scripting/ScriptingOutputCapturer.java PRE-CREATION
>   src/org/apache/pig/scripting/streaming/python/PythonScriptEngine.java PRE-CREATION
>   src/python/streaming/controller.py PRE-CREATION
>   src/python/streaming/pig_util.py PRE-CREATION
>   test/org/apache/pig/impl/builtin/TestStreamingUDF.java PRE-CREATION
>   test/org/apache/pig/impl/streaming/TestExecutableManager.java 6246019
>   test/org/apache/pig/impl/streaming/TestPigToStreamUDF.java PRE-CREATION
>   test/org/apache/pig/impl/streaming/TestStreamUDFToPig.java PRE-CREATION
>   test/org/apache/pig/impl/streaming/TestStreamingUDFOutputHandler.java PRE-CREATION
>   test/org/apache/pig/impl/streaming/TestStreamingUtil.java PRE-CREATION
>   test/org/apache/pig/test/TestPigStreaming.java PRE-CREATION
>   test/python/streaming/test_controller.py PRE-CREATION
>   test/unit-tests d52ad9d
>
> Diff: https://reviews.apache.org/r/13781/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jeremy Karn
>
>