Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Review Request: [PIG-3173] - Partition filter pushdown does not happen if partition keys condition include a AND and OR construct


+
Rohini Palaniswamy 2013-03-20, 00:16
+
Dmitriy Ryaboy 2013-03-20, 00:42
Copy link to this message
-
Re: Review Request: [PIG-3173] - Partition filter pushdown does not happen if partition keys condition include a AND and OR construct
Dmitriy Ryaboy 2013-03-20, 19:45


> On March 20, 2013, 12:42 a.m., Dmitriy Ryaboy wrote:
> > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/newplan/PColFilterExtractor.java, line 224
> > <https://reviews.apache.org/r/10035/diff/1/?file=272254#file272254line224>
> >
> >     (A and B) or (C and D)
> >    
> >     is impossible if (A or C) is false. We can push this up, while retaining the original filter to apply the original filter.
>
> Rohini Palaniswamy wrote:
>     So just to confirm, you want to extract A and C from each AND condition and push (A OR C) as the partition filter for optimization and still leave ((A AND B) or (C AND D)) to be applied on each tuple?

correct, unless my logic is wrong.

I actually think we made a bad decision when we decided that if we can push partitions down, we can drop the filter on the pig side -- this means we can't take advantage of partial filters loaders might support (for example, a bloom filter a loader can consult to return just the rows that "probably" match the condition, as opposed to definitely match. With filter removal, we have to have loaders implement a second-pass filtering on top of such filters).
- Dmitriy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10035/#review18127
-----------------------------------------------------------
On March 20, 2013, 12:16 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10035/
> -----------------------------------------------------------
>
> (Updated March 20, 2013, 12:16 a.m.)
>
>
> Review request for pig.
>
>
> Description
> -------
>
> 1) Fixed cases where partition pushdown was not happening for AND and OR construct
> 2) Commented out the negative test cases as they were actually not asserting anything.
>
>
> This addresses bug PIG-3173.
>     https://issues.apache.org/jira/browse/PIG-3173
>
>
> Diffs
> -----
>
>   http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/newplan/PColFilterExtractor.java 1458047
>   http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestPartitionFilterPushDown.java 1458047
>
> Diff: https://reviews.apache.org/r/10035/diff/
>
>
> Testing
> -------
>
> Unit tests added and tested few cases manually with hcat.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>

+
Rohini Palaniswamy 2013-03-20, 19:02
+
Rohini Palaniswamy 2013-04-29, 20:22
+
Cheolsoo Park 2013-04-29, 20:51