Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill, mail # dev - Expressions in the Logical Plan


Copy link to this message
-
Re: Expressions in the Logical Plan
Julian Hyde 2012-11-18, 21:02
On Nov 17, 2012, at 1:13 PM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:

> Let me give an answer that may help understanding.  The logical plan tries
> to be as generic as possible.  As such, all types are basic types.  That
> being said, we'll surely have a large selection of various types of
> functions which could modify a value.  For example currency(integer) might
> convert a number of pennies into a dollar amount 3500 > "$35.00".  The
> query layer will be responsible for converting the particular query
> language to a set of these various functions.  Then the optimizer decides
> where to apply those functions.  Lastly, the execution layer runs them as
> efficiently as possible givens it execution environment.  Given all of
> this, it is highly likely (especially early on) that a new parser may also
> introduce new functions that need to be supported at the execution layer.

Sorry I've been quiet on the list recently.

I would be delighted to help Drill use the Optiq framework. It has all of this stuff, is proven and extensible.

Specially regarding expressions. Take a look at RexProgram. It models a collection of expressions required to implement a project ("select clause") and a filter. RexProgram is a convenient base for common-subexpression elimination, shortcut generation, and so forth.

Some of the finer details: There is an array of expressions, topologically sorted such that each expression is based only on expressions earlier in the array. Expressions are either inputs, literals, or calls to functions; functions can have multiple arguments, but the arguments must be references to expressions earlier in the array. Expressions are very strongly typed; casting which might be implicit in the query has been made explicit, so arguments have exactly the type required by the function.  Although it is a java class, it would easily serialize to a form similar to a text form similar to the examples in this thread.

https://github.com/julianhyde/optiq/blob/master/src/main/java/org/eigenbase/rex/RexProgram.java

I've been busy on my book and other projects, and I remain too busy to take ownership for of a Drill component, but I would be happy to help someone else evaluate Optiq (or pieces of Optiq) for use in Drill.

Julian