Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill, mail # dev - Re: [jira] [Updated] (DRILL-60) multiple query support


Copy link to this message
-
Re: [jira] [Updated] (DRILL-60) multiple query support
Lisen Mu 2013-07-30, 17:24
Jacques,

The review board got me. will try tomorrow.

This piece of work is very useful in our case. Thanks in advance for any
comments!

On Wed, Jul 31, 2013 at 1:15 AM, Lisen Mu (JIRA) <[EMAIL PROTECTED]> wrote:

>
>      [
> https://issues.apache.org/jira/browse/DRILL-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Lisen Mu updated DRILL-60:
> --------------------------
>
>     Attachment: DRILL-60.0.patch.txt
>
> The review board complains about
>
> 'The file
> 'sandbox/prototype/exec/java-exec/src/main/codegen/ValueVectors/templates/FixedValueVectors.java'
> (r13fa5eb) could not be found in the repository'
>
> I would deal with review board tomorrow. please see the attachment first.
>
> union pop related commits included, for a test case that finally makes
> sense.
>
>
>
> > multiple query support
> > ----------------------
> >
> >                 Key: DRILL-60
> >                 URL: https://issues.apache.org/jira/browse/DRILL-60
> >             Project: Apache Drill
> >          Issue Type: New Feature
> >            Reporter: Lisen Mu
> >         Attachments: DRILL-60.0.patch.txt
> >
> >
> > parse/optimize several SQL(or other QL)s into one logical plan and
> execute them once.
> >
> > * use case
> > If a report generating system is used as a frontend system of drill
> instead of a command line, There would be a set of similar SQL to execute
> to generate a report(consider each day's DAU statistics with geoip
> distribution(groupby) for 7 continuous days). If processed individually,
> plans for SQLs in this kind of set are similar(scan pv log in these days)
> or same(scan GeoIP table) in majority. If processed together, duplicated
> scan(GeoIP) could be eliminated to reduce I/O, or similar scan(pv) could be
> merged to achieve better throughput, and provides chances of other
> intermediate result reuse.
> > * possibility
> > Query in a QL is represented as a tree. Drill models logical plan and
> physical plan as DAG, which is more expressive than tree. More precisely, a
> set of trees(with common nodes or common edges) can be union-ed into a dag.
> So, the current drill logical/physical plan model supports multiple query
> already. Each sql would have a corresponding StoreOP as its result.
> > * problems
> > To do so, physical operators must support multiple iteration over its
> output. That would have side effect on mem management.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>