Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request 14552: PIG-3480 TFile-based tmpfile compression crashes in some cases


Copy link to this message
-
Re: Review Request 14552: PIG-3480 TFile-based tmpfile compression crashes in some cases

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14552/#review26948
-----------------------------------------------------------

trunk/conf/pig.properties
<https://reviews.apache.org/r/14552/#comment52481>

    not your change, but now that I see it -- typo in the parameter name

trunk/conf/pig.properties
<https://reviews.apache.org/r/14552/#comment52484>

    drop -- this is redefined below

trunk/conf/pig.properties
<https://reviews.apache.org/r/14552/#comment52482>

    add a comment that says what the possible values are?

trunk/conf/pig.properties
<https://reviews.apache.org/r/14552/#comment52483>

    update

trunk/src/org/apache/pig/PigConfiguration.java
<https://reviews.apache.org/r/14552/#comment52485>

    just for consistency, can we make TFile take 'gz' or 'gzip' while we are in here?

trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java
<https://reviews.apache.org/r/14552/#comment52492>

    the name is a bit misleading -- can you rename it so it's more clear that this is copying tmp file compression settings only?
    
    what about the tmpfile compression storage config?

trunk/src/org/apache/pig/impl/io/TFileStorage.java
<https://reviews.apache.org/r/14552/#comment52494>

    let's allow "gzip" and rewrite it to gz
Looks mostly right! Just a few superficial comments.

- Dmitriy Ryaboy
On Oct. 11, 2013, 8:43 p.m., Aniket Mokashi wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14552/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2013, 8:43 p.m.)
>
>
> Review request for pig, Cheolsoo Park, Dmitriy Ryaboy, Julien Le Dem, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3480
>     https://issues.apache.org/jira/browse/PIG-3480
>
>
> Repository: pig
>
>
> Description
> -------
>
> - Added a new parameter to make SequenceFileInterStorage optional.
> - Added tests
> - Refactored apis
>
>
> Diffs
> -----
>
>   trunk/conf/pig.properties 1530468
>   trunk/src/org/apache/pig/PigConfiguration.java 1530468
>   trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java 1530468
>   trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 1530468
>   trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartitionRearrange.java 1530468
>   trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java 1530468
>   trunk/src/org/apache/pig/impl/io/InterStorage.java 1530468
>   trunk/src/org/apache/pig/impl/io/SequenceFileInterStorage.java PRE-CREATION
>   trunk/src/org/apache/pig/impl/io/TFileStorage.java 1530468
>   trunk/src/org/apache/pig/impl/util/Utils.java 1530468
>   trunk/test/org/apache/pig/test/TestTmpFileCompression.java 1530468
>
> Diff: https://reviews.apache.org/r/14552/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aniket Mokashi
>
>