Cheolsoo Park 2013-10-11, 21:56
-Re: Review Request 14601: PIG-3502 Initial implementation of TezLauncher
Cheolsoo Park 2013-10-12, 01:19
This is an automatically generated e-mail. To reply, visit:
(Updated Oct. 12, 2013, 1:19 a.m.)
Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
Looking at the mini cluster log, I found that DAG was failing with NPE because local resources were not set for each vertex. I updated DAG construction code to set local resources in vertex.
Now DAG is successfully launched by AM, but it fails due to YARN-1229 in hadoop 2.1.1-beta. The test case doesn't fail because it doesn't check the job status for now. I will follow up with Tez devs about this YARN AM failure.
Btw, there was an incorrect statement in my description:
>> Removes kill() method from ExecutionEngine interface. It was used merely by Launcher, and it didn't seem useful.
kill() was used by ExecutionEngineKiller, but since I moved it to Launcher and renamed it as HangingJobKiller, kill() is no longer needed in ExecutionEngine interface.
Initial implementation of TezLauncher. The patch includes the following changes:
- Refactors Launcher and MapReduceLauncher. Basically, I moved what's MR-specific to MapReduceLauncher and kept what's common in Launcher.
- Removes kill() method from ExecutionEngine interface. It was used merely by Launcher, and it didn't seem useful.
- Implements the DAG submission of TezLauncher. Basically, I overrode the submit() method in TezJob (extension of ControlledJob) that calls DAGClient.submit().
- Adds a skeleton TezStats class that extends PigStats. This class needs to be implemented to monitor job status after submission.
- Adds a unit test called TestTezLauncher. For now, all it does is to compile queries and submit a DAG. More meaningful unit tests will be added after TezStats is implemented.
- Adds a TezLocalExecType. To start up a YARN mini cluster, I needed non-local Tez mode. So I created "tez_local" and "tez" modes.
- Whitespace and indentation fixes.
TestTezLauncher is added.
Mark Wagner 2013-10-12, 05:30