Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # dev >> non map-reduce for simple queries


+
Namit Jain 2012-07-28, 20:35
Copy link to this message
-
Re: non map-reduce for simple queries
If where condition is too complex , selecting specific columns seems simple
enough and useful.

On Saturday, July 28, 2012, Namit Jain <[EMAIL PROTECTED]> wrote:
> Currently, hive does not launch map-reduce jobs for the following queries:
>
> select * from <T> where <condition on partition columns> (limit <n>)?
>
> This behavior is not configurable, and cannot be altered.
>
> HIVE-2925 wants to extend this behavior. The goal is not to spawn
map-reduce jobs for the following queries:
>
> Select <expr> from <T> where <any condition> (limit <n>)?
>
> It is currently controlled by one parameter:
hive.aggressive.fetch.task.conversion, based on which it is decided,
whether to spawn
> map-reduce jobs or not for the queries of the above type. Note that this
can be beneficial for certain types of queries, since it is
> avoiding the expensive step of spawning map-reduce. However, it can be
pretty expensive for certain types of queries: selecting
> a very large number of rows, the query having a very selective filter
(which is satisfied by a very number of rows, and therefore involves
> scanning a very large table) etc. The user does not have any control on
this. Note that it cannot be done by hooks, since the pre-semantic
> hooks does not have enough information: type of the query, inputs etc.
and it is too late to do anything in the post-semantic hook (the
> query plan has already been altered).
>
> I would like to propose the following configuration parameters to control
this behavior.
> hive.fetch.task.conversion: true, false, auto
>
> If the value is true, then all queries with only selects and filters will
be converted
> If the value is false, then no query will be converted
> If the value is auto (which should be the default behavior), there should
be additional parameters to control the semantics.
>
> hive.fetch.task.auto.limit.threshold               ---> integer value X1
> hive.fetch.task.auto.inputsize.threshold      ---> integer value X2
>
> If either the query has a limit lower than X1, or the input size is
smaller than X2, the queries containing only filters and selects will be
converted to not use
> map-reudce jobs.
>
>
> Comments…
>
> -namit
>
>
>
+
Navis류승우 2012-07-29, 01:17
+
Namit Jain 2012-07-29, 14:28
+
Namit Jain 2012-07-29, 14:45
+
Owen OMalley 2012-07-30, 21:01
+
Navis류승우 2012-07-31, 01:37
+
Namit Jain 2012-07-31, 04:12
+
Owen OMalley 2012-07-31, 06:31
+
Namit Jain 2012-07-31, 06:38
+
Owen OMalley 2012-07-31, 15:53
+
Namit Jain 2012-07-31, 17:47
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB