Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> contribution


Copy link to this message
-
Re: contribution
Hey Jacques

Sorry to be a nag, but is there any change to take a sneak peak at the protobuf rpc stuff?
I'd really like hack something together wrt to the daemon this weekend.
Also, wrt to configuration management (zk/helix) maybe you could post the iface so that it'd be possible to hack something static (i.e. non-ft, properties file based) just to make dist execution work.

Thanks
David

On Mar 16, 2013, at 8:34 PM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:

> Hey David,
>
> The java-exec framework is not far enough along that it makes sense for me
> to push it externally yet.  However, I did push my initial wip physical
> plan approach.  You can find it here:
> https://github.com/jacques-n/incubator-drill/tree/physical_plan_updates
>
> Hopefully, I will get further along on the java-exec stuff soon.
>
> I'd suggest that you focus your energy on the StorageEngine API and HBase
> implementation.  If you're up for it, let's do a quick skype chat to sync
> up.  Let me know your availability over the next few days.
>
> Thanks,
> Jacques
>
>
>
> On Fri, Mar 15, 2013 at 6:59 PM, David Alves <[EMAIL PROTECTED]> wrote:
>
>> that'd be great thanks.
>>
>> -david
>>
>> On Mar 15, 2013, at 8:51 PM, Jacques Nadeau <[EMAIL PROTECTED]>
>> wrote:
>>
>>> I've been under the weather the last few days and haven't made much
>>> progress. Let me see if I can get you something tomorrow.
>>>
>>> On Mar 15, 2013, at 2:36 PM, David Alves <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Jacques
>>>>
>>>>  Is there any chance we could get a preview of this physical plan
>> stuff and basic plumbing for distributed execution before the weekend?
>> maybe in a github branch somewhere?
>>>>  I mean it doesn't have to be complete or even running, I'd just like
>> to make some progress with other stuff and keeping it in line with
>> whichever plumbing you already have would be great.
>>>>
>>>> Best
>>>> David
>>>>
>>>> On Mar 13, 2013, at 3:12 PM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> I'm working on some physical plan stuff as well as some basic plumbing
>> for
>>>>> distributed execution.  Its very in progress so I need to clean things
>> up a
>>>>> bit before we could collaborate/ divide and conquer on it.  Depending
>> on
>>>>> your timing and availability, maybe I could put some of this together
>> in
>>>>> the next couple days so that you could plug in rather than reinvent.
>> In
>>>>> the meantime, pushing forward the builder stuff, additional test cases
>> on
>>>>> the reference interpreter and/or thinking through the logical plan
>> storage
>>>>> engine pushdown/rewrite could be very useful.
>>>>>
>>>>> Let me know your thoughts.
>>>>>
>>>>> thanks,
>>>>> Jacques
>>>>>
>>>>> On Wed, Mar 13, 2013 at 9:47 AM, David Alves <[EMAIL PROTECTED]>
>> wrote:
>>>>>
>>>>>> Hi Jacques
>>>>>>
>>>>>>     I can assign issues to me now, thanks.
>>>>>>     What you say wrt to the logical/physical/execution layers sounds
>>>>>> good.
>>>>>>     My main concern, for the moment is to have something working as
>>>>>> fast as possible, i.e. some daemons that I'd be able to deploy to a
>> working
>>>>>> hbase cluster and send them work to do in some form (first step would
>> be to
>>>>>> treat is as a non distributed engine where each daemon runs an
>> instance of
>>>>>> the prototype).
>>>>>>     Here's where I'd like to go next:
>>>>>>     - lay the ground work for the daemons (scripts/rpc iface/wiring
>>>>>> protocol).
>>>>>>     - create an execution engine iface that allows to abstract future
>>>>>> implementations, and make it available through the rpc iface. this
>> would
>>>>>> sit in front of the ref impl for now and would be replaced by cpp
>> down the
>>>>>> line.
>>>>>>
>>>>>>     I think we can probably concentrate on the capabilities iface a
>>>>>> bit down the line but, as a first approach, I see it simply providing
>> a
>>>>>> simple set of ops that it is able to run internally.
>>>>>>     How to abstract locality/partitioning/schema capabilities is till
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB