Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill, mail # dev - B[yi]teSize execwork tasks someone could potentially help out with...


Copy link to this message
-
Re: B[yi]teSize execwork tasks someone could potentially help out with...
Timothy Chen 2013-04-26, 17:04
Jacques I think this is the one I emailed you before that has no licensing info.

Tim

Sent from my iPhone

On Apr 26, 2013, at 9:30 AM, David Alves <[EMAIL PROTECTED]> wrote:

> i've looked through it and looks like it can leverage shared memory, which I was looking for anyway.
> I also like the way garbage collection works (gc in java also clears off-heap).
> I'll take a deeper look during the weekend.
>
> -david
>
> On Apr 26, 2013, at 11:25 AM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:
>
>> I've looked at that in the past and think the idea of using here is very
>> good.  It seems like ByteBuf is nice as it has things like endianess
>> capabilities, reference counting and management and Netty direct support.
>> On the flipside, larray is nice for its large array capabilities and
>> better input/output interfaces.  The best approach might be to define a new
>> ByteBuf implementation that leverages LArray.  I'll take a look at this in
>> a few days if someone else doesn't want to.
>>
>> j
>>
>> On Fri, Apr 26, 2013 at 8:39 AM, kishore g <[EMAIL PROTECTED]> wrote:
>>
>>> Fort *ByteBuf Improvements*, Have you looked at LArrayJ
>>> https://github.com/xerial/larray. It has those wrappers and I found it
>>> quite useful. The same person has also written java version for snappy
>>> compression. Not sure if you guys have plan to add compression, but one of
>>> the nice things I could do was use the memory offsets for source(compressed
>>> data) and dest(uncompressed array) and do the decompression off-heap. It
>>> supports the need for looking up by index and has wrappers for most of the
>>> primitive data types.
>>>
>>> Are you looking at something like this?
>>>
>>> thanks,
>>> Kishore G
>>>
>>>
>>>
>>> On Fri, Apr 26, 2013 at 7:53 AM, Jacques Nadeau <[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>> They are on the list but the list is long :)
>>>>
>>>> Have a good weekend.
>>>>
>>>> On Thu, Apr 25, 2013 at 9:51 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> So if no one picks anything up you will be done with all the work in
>>> the
>>>>> next couple of days? :)
>>>>>
>>>>> Would like to help out but I'm traveling to la over the weekend.
>>>>>
>>>>> I'll sync with you Monday to see how I can help then.
>>>>>
>>>>> Tim
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On Apr 25, 2013, at 9:06 PM, Jacques Nadeau <[EMAIL PROTECTED]>
>>> wrote:
>>>>>
>>>>>> I'm working on the execwork stuff and if someone would like to help
>>>> out,
>>>>>> here are a couple of things that need doing.  I figured I'd drop them
>>>>> here
>>>>>> and see if anyone wants to work on them in the next couple of days.
>>> If
>>>>> so,
>>>>>> let me know otherwise I'll be picking them up soon.
>>>>>>
>>>>>> *RPC*
>>>>>> - RPC Layer Handshakes: Currently, I haven't implemented the
>>> handshake
>>>>> that
>>>>>> should happen in either the User <> Bit or the Bit <> Bit layer.  The
>>>>> plan
>>>>>> was to use an additional inserted event handler that removed itself
>>>> from
>>>>>> the event pipeline after a successful handshake or disconnected the
>>>>> channel
>>>>>> on a failed handshake (with appropriate logging).  The main
>>> validation
>>>> at
>>>>>> this point will be simply confirming that both endpoints are running
>>> on
>>>>> the
>>>>>> same protocol version.   The only other information that is currently
>>>>>> needed is that that in the Bit <> Bit communication, the client
>>> should
>>>>>> inform the server of its DrillEndpoint so that the server can then
>>> map
>>>>> that
>>>>>> for future communication in the other direction.
>>>>>>
>>>>>> *DataTypes*
>>>>>> - General Expansion: Currently, we have a hodgepodge of datatypes
>>>> within
>>>>>> the org.apache.drill.common.expression.types.DataType.  We need to
>>>> clean
>>>>>> this up.  There should be types that map to standard sql types.  My
>>>>>> thinking is that we should actually have separate types for each for
>>>>>> nullable, non-nullable and repeated (required, optional and repeated