Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request: PIG-3288 Kill jobs if the number of output files is over a configurable limit


Copy link to this message
-
Review Request: PIG-3288 Kill jobs if the number of output files is over a configurable limit
Cheolsoo Park 2013-06-10, 16:17

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11719/
-----------------------------------------------------------

Review request for pig.
Description
-------

 Adds a new property called pig.exec.hdfs.files.max.limit via which the user can specify the max number of created files by MR jobs. If it exceeds the configured limit, Pig kills running jobs and exit.
This addresses bug PIG-3288.
    https://issues.apache.org/jira/browse/PIG-3288
Diffs
-----

  conf/pig.properties 8df5bea
  src/docs/src/documentation/content/xdocs/admin.xml d8febd4
  src/org/apache/pig/PigConfiguration.java b94e0c8
  src/org/apache/pig/PigCounters.java 1249774
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java f06b90a
  test/e2e/pig/tests/negative.conf 64465af
  test/e2e/pig/udfs/java/org/apache/pig/test/udf/storefunc/PigStorageWithFileCount.java e69de29

Diff: https://reviews.apache.org/r/11719/diff/
Testing
-------

All unit tests pass.

A new e2e test case is added for the new property.
Thanks,

Cheolsoo Park