Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> Questions about Optiq

Copy link to this message
Re: Questions about Optiq
On Dec 4, 2012, at 5:54 PM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:

>   1. Do you have thoughts on whether we should fork Optiq or depend on it?
>    (It seems like it still has a lot of stuff tied to execution within it…)

I think you should depend on it. I build using maven, so it is easy for me to deploy jar files with a pom.xml of their dependencies.

There is stuff inside Optiq that needs to be factored out (into another optiq-xxx project, or removed entirely). Use the public APIs, ignore the other stuff, and it will be gone soon enough.

>   2. Any thoughts on how well exchange node fits in for parallelization?
>    I suppose we could define a new ExchangeRel and number of associated
>   rules.  But we’d also need to implement some reducing/affecting cascade for
>   costs of subsequent steps that are parallelized.  I suppose that it really
>   goes to whether we’re optimizing latency or total machine utilization.

Yes, we could introduce ExchangeRel to optimize both parallelism on the same machine and for distribution. "Core" and "Node" could possibly be traits (i.e. physical properties of the intermediate query result) and therefore ExchangeRel would be in some sense a converter.

Optimizing parallelism is challenging, because one would not want to create a sub-expression for every core of every node in a cluster (8 cores x 1000 nodes = 8000 RelNodes in the plan!). This problem has been studied extensively, and I am confident that Optiq can be used to build the solutions proposed by the literature.

>   3. It looks like Optiq supports the concept of multiple orderings for
>   the same data.  However, it doesn’t seen to support things such as
>   preexisting data partitions or multiple subprojections (e.g. different
>   collection of fields with different orderings).  I would generally think of
>   these as physical properties of the underlying data.  It seems like that is
>   what RelTraits are for…?

RelTraits are physical properties of the RelNode (i.e. the intermediate query result). By definition, a physical property is something that can be changed (e.g. you can sort to change the ordering property; you can transmit over the network to change the location property; you can use a bridge to change the calling-convention property).

Partitions and projections do not meet that definition of physical properties. (Although each data set from a partition/projection has a physical location.) I would deal with them as specific cases of materialized views.

For example, suppose you have

Table EMPS
Partition FEMALE_EMPS: select * from EMPS where gender = 'F'
Sorted projection EMP_DEPTS: select empno, deptno from EMPS order by deptno

You would create an initial planner graph of

Set 0:
  Subset 0.0 (sort={})

Set 1:
  Subset 1.0 (sort={})
    Filter(EMPS, gender = 'F')

Set 2:
  Subset 2.0 (sort={(deptno)})
    Sort(subset 2.1, {deptno})
  Subset 2.1 (sort={})
     Project(subset 0.0, {empno, deptno})

Note that:
* Sets are equivalence sets. Rels in the same set always yield the same results.
* Subsets are rels that are equivalent and have the same physical properties. E.g. those in subset 2.0 are all sorted by deptno.

Given this initial state, the if someone asks for "select distinct deptno from emps where gender = 'F'" initial query will be based on set 0, but the planner will be able to find a way to use subset 2.0.

>   4. RelTraits seem like they were envisioned as a “superclass” of
>   physical properties.  However, the actual implementation seems to only be
>   utilized for CallingConvention.   Am I missing something here?

Projects that have used optiq have added other kinds of traits. I can't discuss the details of these. CallingConvention is the only trait that is left in optiq core (and maybe that too would go, if as you suggest, I remove the "stuff tied to execution").

Traits are orthogonal. E.g. if location and ordering are traits, then we might have one subset with (location=Node1, ordering={deptno}), another subset with (location=Node1, ordering={empno}), and another subset with (location=Node2, ordering={empno}).
Currently we treat ordering as a logical property. This is an artifact of how Optiq was used for optimizing stream processing; you can't sort a whole stream (because it is infinite), therefore we treated order as a given. If we change ordering to be a physical property, which is more conventional, I think we can implement converters to give required orderings.
This comes from Saffron. My goal was to implement relational algebra on top of Java collections. Vectors, Arrays, Lists, Iterators are all forms of the same data, and there is a cost to convert between them. E.g. you can convert an array to an iterator by generating Arrays.asList(a).iterator().

It sounds like Drill will have Java & C as calling conventions, and maybe variants such as JDBC.

However, calling conventions don't need to be baked in. That's why CallingConvention is not an enum. You can provide an instance of CallingConvention that is not defined as a constant in that class. That's not very obvious from the code right now. I think I'll make CallingConvention into an interface, and move the pre-defined instances out of core Optiq.
Obviously, early binding makes things easier. For one thing, SQL is basically an early-binding language; and things are more predictable if you are working with records that have fixed fields, known types, and known maximum lengths.

But there are strategies to deal with late-bound types.

First, capitalize on any early-binding type information you have. If you know every record has an id field, then model records as (id: long, others: map).

Second, SQL can be extended to be late binding. In some cases, it's just syntactic sugar. If I write 'select id, starSign from emp', it would be converted to 'select id, others.get('starSign') from emp'.

For the Splunk adapter I was ab
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB