Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request 14776: PIG-3514 Initial implementation of TezStats


Copy link to this message
-
Re: Review Request 14776: PIG-3514 Initial implementation of TezStats

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14776/#review27266
-----------------------------------------------------------

Ship it!

src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java
<https://reviews.apache.org/r/14776/#comment53114>

    We will soon need to launch multiple jobs from Pig, and collect and aggregate multiple TezStats.
Looks good generally. With the patch, some e2e tests runs successfully. Let's commit it and go from there.

- Daniel Dai
On Oct. 20, 2013, 11:06 p.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14776/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2013, 11:06 p.m.)
>
>
> Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3514
>     https://issues.apache.org/jira/browse/PIG-3514
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> This is an initial implementation of TezStats. For now, it collects the number of succeeded/failed Tez vertices and prints the "success/failed" message at the end.
>
> In summary, I implemented the following classes:
> * TezStats extends PigStats
> * TezVertexStats extends JobStats
>
> Note that TezVertexStats captures a Tez vertex not a Tez job.
>
> In addition, I moved several fields and methods that can be commonly used by both SimplePigStats and TezStats to PigStats.
>
>
> Diffs
> -----
>
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecType.java c726923
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecutionEngine.java 6e748a8
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java d88eb6b
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java d7577c1
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezScriptState.java eb4eefb
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezStats.java ef8733a
>   src/org/apache/pig/tools/pigstats/PigStats.java 6d2e58e
>   src/org/apache/pig/tools/pigstats/mapreduce/SimplePigStats.java bbfd5a9
>   src/org/apache/pig/tools/pigstats/tez/TezScriptState.java e69de29
>   src/org/apache/pig/tools/pigstats/tez/TezStats.java e69de29
>   src/org/apache/pig/tools/pigstats/tez/TezTaskStats.java e69de29
>   test/org/apache/pig/tez/TestTezLauncher.java 0a3bc73
>
> Diff: https://reviews.apache.org/r/14776/diff/
>
>
> Testing
> -------
>
> Ran a MRR job on a single node Tez cluster and confirmed that the job status is printed correctly. Here are examples:
>
> -----
> 2013-10-20 00:29:13,218 [main] INFO  org.apache.pig.tools.pigstats.tez.TezStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 2.2.0 0.13.0-SNAPSHOT cheolsoop 2013-10-20 00:28:44 2013-10-20 00:29:13 GROUP_BY,FILTER
>
> Success!
> -----
> 2013-10-20 00:30:10,970 [main] INFO  org.apache.pig.tools.pigstats.tez.TezStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 2.2.0 0.13.0-SNAPSHOT cheolsoop 2013-10-20 00:29:30 2013-10-20 00:30:10 GROUP_BY,FILTER
>
> Failed!
> -----
>
> More unit tests will be added after Tez mini cluster is added.
>
>
> Thanks,
>
> Cheolsoo Park
>
>