Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Filtering


Copy link to this message
-
Re: Filtering
Owen O'Malley 2013-05-15, 17:35
On Wed, May 15, 2013 at 3:38 AM, Peter Marron <
[EMAIL PROTECTED]> wrote:

>  Hi,****
>
> ** **
>
> I’m using Hive 0.10.0 and Hadoop 1.0.4.****
>
> ** **
>
> I would like to create a normal table but have some of my code run so that
> I can remove filtering****
>
> parts of the query and limit the output in the splits of the InputFormat.
> I believe that this is****
>
> “Filtering Pushdown” as described in
> https://cwiki.apache.org/Hive/filterpushdowndev.html****
>
> I have tried various approaches and run into problems and I was wondering*
> ***
>
> if anyone had any suggestions as to how I might proceed.
>
I've started doing similar work for the ORC reader.

> ****
>
> ** **
>
> Firstly although that page mentions InputFormat there doesn’t seem to be
> any way (that I can find)****
>
> to perform filter passing to InputFormats and so I gave up on that
> approach.
>

There is. You just need to set  hive.optimize.index.filter to true. See
https://issues.apache.org/jira/browse/HIVE-4242.

That said, we really need to create a better interface that allows
inputformats to negotiate what parts of the predicate they can process.

-- Owen

> **
>