Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Filtering


On Wed, May 15, 2013 at 3:38 AM, Peter Marron <
[EMAIL PROTECTED]> wrote:

>  Hi,****
>
> ** **
>
> I’m using Hive 0.10.0 and Hadoop 1.0.4.****
>
> ** **
>
> I would like to create a normal table but have some of my code run so that
> I can remove filtering****
>
> parts of the query and limit the output in the splits of the InputFormat.
> I believe that this is****
>
> “Filtering Pushdown” as described in
> https://cwiki.apache.org/Hive/filterpushdowndev.html****
>
> I have tried various approaches and run into problems and I was wondering*
> ***
>
> if anyone had any suggestions as to how I might proceed.
>
I've started doing similar work for the ORC reader.

> ****
>
> ** **
>
> Firstly although that page mentions InputFormat there doesn’t seem to be
> any way (that I can find)****
>
> to perform filter passing to InputFormats and so I gave up on that
> approach.
>

There is. You just need to set  hive.optimize.index.filter to true. See
https://issues.apache.org/jira/browse/HIVE-4242.

That said, we really need to create a better interface that allows
inputformats to negotiate what parts of the predicate they can process.

-- Owen

> **
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB