Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill, mail # dev - contribution


Copy link to this message
-
Re: contribution
David Alves 2013-03-22, 19:06
Hey Jacques

Sorry to be a nag, but is there any change to take a sneak peak at the protobuf rpc stuff?
I'd really like hack something together wrt to the daemon this weekend.
Also, wrt to configuration management (zk/helix) maybe you could post the iface so that it'd be possible to hack something static (i.e. non-ft, properties file based) just to make dist execution work.

Thanks
David

On Mar 16, 2013, at 8:34 PM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:

> Hey David,
>
> The java-exec framework is not far enough along that it makes sense for me
> to push it externally yet.  However, I did push my initial wip physical
> plan approach.  You can find it here:
> https://github.com/jacques-n/incubator-drill/tree/physical_plan_updates
>
> Hopefully, I will get further along on the java-exec stuff soon.
>
> I'd suggest that you focus your energy on the StorageEngine API and HBase
> implementation.  If you're up for it, let's do a quick skype chat to sync
> up.  Let me know your availability over the next few days.
>
> Thanks,
> Jacques
>
>
>
> On Fri, Mar 15, 2013 at 6:59 PM, David Alves <[EMAIL PROTECTED]> wrote:
>
>> that'd be great thanks.
>>
>> -david
>>
>> On Mar 15, 2013, at 8:51 PM, Jacques Nadeau <[EMAIL PROTECTED]>
>> wrote:
>>
>>> I've been under the weather the last few days and haven't made much
>>> progress. Let me see if I can get you something tomorrow.
>>>
>>> On Mar 15, 2013, at 2:36 PM, David Alves <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Jacques
>>>>
>>>>  Is there any chance we could get a preview of this physical plan
>> stuff and basic plumbing for distributed execution before the weekend?
>> maybe in a github branch somewhere?
>>>>  I mean it doesn't have to be complete or even running, I'd just like
>> to make some progress with other stuff and keeping it in line with
>> whichever plumbing you already have would be great.
>>>>
>>>> Best
>>>> David
>>>>
>>>> On Mar 13, 2013, at 3:12 PM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> I'm working on some physical plan stuff as well as some basic plumbing
>> for
>>>>> distributed execution.  Its very in progress so I need to clean things
>> up a
>>>>> bit before we could collaborate/ divide and conquer on it.  Depending
>> on
>>>>> your timing and availability, maybe I could put some of this together
>> in
>>>>> the next couple days so that you could plug in rather than reinvent.
>> In
>>>>> the meantime, pushing forward the builder stuff, additional test cases
>> on
>>>>> the reference interpreter and/or thinking through the logical plan
>> storage
>>>>> engine pushdown/rewrite could be very useful.
>>>>>
>>>>> Let me know your thoughts.
>>>>>
>>>>> thanks,
>>>>> Jacques
>>>>>
>>>>> On Wed, Mar 13, 2013 at 9:47 AM, David Alves <[EMAIL PROTECTED]>
>> wrote:
>>>>>
>>>>>> Hi Jacques
>>>>>>
>>>>>>     I can assign issues to me now, thanks.
>>>>>>     What you say wrt to the logical/physical/execution layers sounds
>>>>>> good.
>>>>>>     My main concern, for the moment is to have something working as
>>>>>> fast as possible, i.e. some daemons that I'd be able to deploy to a
>> working
>>>>>> hbase cluster and send them work to do in some form (first step would
>> be to
>>>>>> treat is as a non distributed engine where each daemon runs an
>> instance of
>>>>>> the prototype).
>>>>>>     Here's where I'd like to go next:
>>>>>>     - lay the ground work for the daemons (scripts/rpc iface/wiring
>>>>>> protocol).
>>>>>>     - create an execution engine iface that allows to abstract future
>>>>>> implementations, and make it available through the rpc iface. this
>> would
>>>>>> sit in front of the ref impl for now and would be replaced by cpp
>> down the
>>>>>> line.
>>>>>>
>>>>>>     I think we can probably concentrate on the capabilities iface a
>>>>>> bit down the line but, as a first approach, I see it simply providing
>> a
>>>>>> simple set of ops that it is able to run internally.
>>>>>>     How to abstract locality/partitioning/schema capabilities is till