Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Drill >> mail # dev >> contribution


+
David Alves 2013-03-11, 10:02
+
Ted Yu 2013-03-11, 16:30
+
Ted Dunning 2013-03-11, 18:43
+
Jacques Nadeau 2013-03-11, 19:13
+
David Alves 2013-03-13, 01:33
+
Lisen Mu 2013-03-13, 04:44
+
David Alves 2013-03-13, 05:19
+
Lisen Mu 2013-03-13, 06:23
+
Jacques Nadeau 2013-03-13, 16:12
+
David Alves 2013-03-13, 16:47
+
Jacques Nadeau 2013-03-13, 20:12
+
David Alves 2013-03-13, 21:30
Copy link to this message
-
Re: contribution
Looking forward to the plumbing as well, since my json scan op sat there
for a while now :)

Tim
On Wed, Mar 13, 2013 at 2:30 PM, David Alves <[EMAIL PROTECTED]> wrote:

> Getting the basic plumbing to a point where we could work together on
> it/use it elsewhere as soon as you can would be awesome.
> As soon as I get that I can start on the daemons/scripts.
> I'll  focus on the SE iface and on HBase pushdown for the moment.
>
> -david
>
> On Mar 13, 2013, at 3:12 PM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:
>
> > I'm working on some physical plan stuff as well as some basic plumbing
> for
> > distributed execution.  Its very in progress so I need to clean things
> up a
> > bit before we could collaborate/ divide and conquer on it.  Depending on
> > your timing and availability, maybe I could put some of this together in
> > the next couple days so that you could plug in rather than reinvent.  In
> > the meantime, pushing forward the builder stuff, additional test cases on
> > the reference interpreter and/or thinking through the logical plan
> storage
> > engine pushdown/rewrite could be very useful.
> >
> > Let me know your thoughts.
> >
> > thanks,
> > Jacques
> >
> > On Wed, Mar 13, 2013 at 9:47 AM, David Alves <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi Jacques
> >>
> >>        I can assign issues to me now, thanks.
> >>        What you say wrt to the logical/physical/execution layers sounds
> >> good.
> >>        My main concern, for the moment is to have something working as
> >> fast as possible, i.e. some daemons that I'd be able to deploy to a
> working
> >> hbase cluster and send them work to do in some form (first step would
> be to
> >> treat is as a non distributed engine where each daemon runs an instance
> of
> >> the prototype).
> >>        Here's where I'd like to go next:
> >>        - lay the ground work for the daemons (scripts/rpc iface/wiring
> >> protocol).
> >>        - create an execution engine iface that allows to abstract future
> >> implementations, and make it available through the rpc iface. this would
> >> sit in front of the ref impl for now and would be replaced by cpp down
> the
> >> line.
> >>
> >>        I think we can probably concentrate on the capabilities iface a
> >> bit down the line but, as a first approach, I see it simply providing a
> >> simple set of ops that it is able to run internally.
> >>        How to abstract locality/partitioning/schema capabilities is till
> >> not clear to me though, thoughts?
> >>
> >> David
> >>
> >> On Mar 13, 2013, at 11:12 AM, Jacques Nadeau <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> I'm working on a presentation that will better illustrate the layers.
> >>> There are actually three key plans.  Thinking to date has been to break
> >>> the plans down into logical, physical and execution.  The third hasn't
> >> been
> >>> expressed well here and is entirely an internal domain to the execution
> >>> engine.  Following some classic methods: Logical expresses what we want
> >> to
> >>> do, Physical expresses how we want to do it (adding points of
> >>> parallelization but not specifying particular amounts of
> parallelization
> >> or
> >>> node by node assignments).  The execution engine is then responsible
> for
> >>> determining the amount of parallelization of a particular plan along
> with
> >>> system load (likely leveraging Berkeley's Sparrow work), task priority
> >> and
> >>> specific data locality information, building sub-dags to be assigned to
> >>> individual nodes and execute the plan.
> >>>
> >>> So in the higher logical and physical levels, a single Scan and
> >> subsequent
> >>> ScanPOP should be okay...  (ScanROPs have a separate problems since
> they
> >>> ignore the level of separation we're planning for the real execution
> >> layer.
> >>> This is the why the current ref impl turns a single Scan into
> potentially
> >>> a union of ScanROPs... not elegant but logically correct.)
> >>>
> >>> The capabilities interface still needs to be defined for how a storage
+
Jacques Nadeau 2013-03-13, 22:25
+
David Alves 2013-03-15, 21:36
+
Jacques Nadeau 2013-03-16, 01:51
+
David Alves 2013-03-16, 01:59
+
Jacques Nadeau 2013-03-17, 01:34
+
David Alves 2013-03-22, 19:06
+
Jacques Nadeau 2013-03-24, 00:13
+
Jacques Nadeau 2013-03-13, 16:14
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB