Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Drill >> mail # dev >> contribution


+
David Alves 2013-03-11, 10:02
+
Ted Yu 2013-03-11, 16:30
+
Ted Dunning 2013-03-11, 18:43
+
Jacques Nadeau 2013-03-11, 19:13
+
David Alves 2013-03-13, 01:33
+
Lisen Mu 2013-03-13, 04:44
+
David Alves 2013-03-13, 05:19
+
Lisen Mu 2013-03-13, 06:23
+
Jacques Nadeau 2013-03-13, 16:12
+
David Alves 2013-03-13, 16:47
+
Jacques Nadeau 2013-03-13, 20:12
+
David Alves 2013-03-13, 21:30
Copy link to this message
-
Re: contribution
Looking forward to the plumbing as well, since my json scan op sat there
for a while now :)

Tim
On Wed, Mar 13, 2013 at 2:30 PM, David Alves <[EMAIL PROTECTED]> wrote:

> Getting the basic plumbing to a point where we could work together on
> it/use it elsewhere as soon as you can would be awesome.
> As soon as I get that I can start on the daemons/scripts.
> I'll  focus on the SE iface and on HBase pushdown for the moment.
>
> -david
>
> On Mar 13, 2013, at 3:12 PM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:
>
> > I'm working on some physical plan stuff as well as some basic plumbing
> for
> > distributed execution.  Its very in progress so I need to clean things
> up a
> > bit before we could collaborate/ divide and conquer on it.  Depending on
> > your timing and availability, maybe I could put some of this together in
> > the next couple days so that you could plug in rather than reinvent.  In
> > the meantime, pushing forward the builder stuff, additional test cases on
> > the reference interpreter and/or thinking through the logical plan
> storage
> > engine pushdown/rewrite could be very useful.
> >
> > Let me know your thoughts.
> >
> > thanks,
> > Jacques
> >
> > On Wed, Mar 13, 2013 at 9:47 AM, David Alves <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi Jacques
> >>
> >>        I can assign issues to me now, thanks.
> >>        What you say wrt to the logical/physical/execution layers sounds
> >> good.
> >>        My main concern, for the moment is to have something working as
> >> fast as possible, i.e. some daemons that I'd be able to deploy to a
> working
> >> hbase cluster and send them work to do in some form (first step would
> be to
> >> treat is as a non distributed engine where each daemon runs an instance
> of
> >> the prototype).
> >>        Here's where I'd like to go next:
> >>        - lay the ground work for the daemons (scripts/rpc iface/wiring
> >> protocol).
> >>        - create an execution engine iface that allows to abstract future
> >> implementations, and make it available through the rpc iface. this would
> >> sit in front of the ref impl for now and would be replaced by cpp down
> the
> >> line.
> >>
> >>        I think we can probably concentrate on the capabilities iface a
> >> bit down the line but, as a first approach, I see it simply providing a
> >> simple set of ops that it is able to run internally.
> >>        How to abstract locality/partitioning/schema capabilities is till
> >> not clear to me though, thoughts?
> >>
> >> David
> >>
> >> On Mar 13, 2013, at 11:12 AM, Jacques Nadeau <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> I'm working on a presentation that will better illustrate the layers.
> >>> There are actually three key plans.  Thinking to date has been to break
> >>> the plans down into logical, physical and execution.  The third hasn't
> >> been
> >>> expressed well here and is entirely an internal domain to the execution
> >>> engine.  Following some classic methods: Logical expresses what we want
> >> to
> >>> do, Physical expresses how we want to do it (adding points of
> >>> parallelization but not specifying particular amounts of
> parallelization
> >> or
> >>> node by node assignments).  The execution engine is then responsible
> for
> >>> determining the amount of parallelization of a particular plan along
> with
> >>> system load (likely leveraging Berkeley's Sparrow work), task priority
> >> and
> >>> specific data locality information, building sub-dags to be assigned to
> >>> individual nodes and execute the plan.
> >>>
> >>> So in the higher logical and physical levels, a single Scan and
> >> subsequent
> >>> ScanPOP should be okay...  (ScanROPs have a separate problems since
> they
> >>> ignore the level of separation we're planning for the real execution
> >> layer.
> >>> This is the why the current ref impl turns a single Scan into
> potentially
> >>> a union of ScanROPs... not elegant but logically correct.)
> >>>
> >>> The capabilities interface still needs to be defined for how a storage
+
Jacques Nadeau 2013-03-13, 22:25
+
David Alves 2013-03-15, 21:36
+
Jacques Nadeau 2013-03-16, 01:51
+
David Alves 2013-03-16, 01:59
+
Jacques Nadeau 2013-03-17, 01:34
+
David Alves 2013-03-22, 19:06
+
Jacques Nadeau 2013-03-24, 00:13
+
Jacques Nadeau 2013-03-13, 16:14