Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> B[yi]teSize execwork tasks someone could potentially help out with...

Copy link to this message
Re: B[yi]teSize execwork tasks someone could potentially help out with...
i've looked through it and looks like it can leverage shared memory, which I was looking for anyway.
I also like the way garbage collection works (gc in java also clears off-heap).
I'll take a deeper look during the weekend.


On Apr 26, 2013, at 11:25 AM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:

> I've looked at that in the past and think the idea of using here is very
> good.  It seems like ByteBuf is nice as it has things like endianess
> capabilities, reference counting and management and Netty direct support.
> On the flipside, larray is nice for its large array capabilities and
> better input/output interfaces.  The best approach might be to define a new
> ByteBuf implementation that leverages LArray.  I'll take a look at this in
> a few days if someone else doesn't want to.
> j
> On Fri, Apr 26, 2013 at 8:39 AM, kishore g <[EMAIL PROTECTED]> wrote:
>> Fort *ByteBuf Improvements*, Have you looked at LArrayJ
>> https://github.com/xerial/larray. It has those wrappers and I found it
>> quite useful. The same person has also written java version for snappy
>> compression. Not sure if you guys have plan to add compression, but one of
>> the nice things I could do was use the memory offsets for source(compressed
>> data) and dest(uncompressed array) and do the decompression off-heap. It
>> supports the need for looking up by index and has wrappers for most of the
>> primitive data types.
>> Are you looking at something like this?
>> thanks,
>> Kishore G
>> On Fri, Apr 26, 2013 at 7:53 AM, Jacques Nadeau <[EMAIL PROTECTED]>
>> wrote:
>>> They are on the list but the list is long :)
>>> Have a good weekend.
>>> On Thu, Apr 25, 2013 at 9:51 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:
>>>> So if no one picks anything up you will be done with all the work in
>> the
>>>> next couple of days? :)
>>>> Would like to help out but I'm traveling to la over the weekend.
>>>> I'll sync with you Monday to see how I can help then.
>>>> Tim
>>>> Sent from my iPhone
>>>> On Apr 25, 2013, at 9:06 PM, Jacques Nadeau <[EMAIL PROTECTED]>
>> wrote:
>>>>> I'm working on the execwork stuff and if someone would like to help
>>> out,
>>>>> here are a couple of things that need doing.  I figured I'd drop them
>>>> here
>>>>> and see if anyone wants to work on them in the next couple of days.
>> If
>>>> so,
>>>>> let me know otherwise I'll be picking them up soon.
>>>>> *RPC*
>>>>> - RPC Layer Handshakes: Currently, I haven't implemented the
>> handshake
>>>> that
>>>>> should happen in either the User <> Bit or the Bit <> Bit layer.  The
>>>> plan
>>>>> was to use an additional inserted event handler that removed itself
>>> from
>>>>> the event pipeline after a successful handshake or disconnected the
>>>> channel
>>>>> on a failed handshake (with appropriate logging).  The main
>> validation
>>> at
>>>>> this point will be simply confirming that both endpoints are running
>> on
>>>> the
>>>>> same protocol version.   The only other information that is currently
>>>>> needed is that that in the Bit <> Bit communication, the client
>> should
>>>>> inform the server of its DrillEndpoint so that the server can then
>> map
>>>> that
>>>>> for future communication in the other direction.
>>>>> *DataTypes*
>>>>> - General Expansion: Currently, we have a hodgepodge of datatypes
>>> within
>>>>> the org.apache.drill.common.expression.types.DataType.  We need to
>>> clean
>>>>> this up.  There should be types that map to standard sql types.  My
>>>>> thinking is that we should actually have separate types for each for
>>>>> nullable, non-nullable and repeated (required, optional and repeated
>> in
>>>>> protobuf vernaciular) since we'll generally operate with those values
>>>>> completely differently (and that each type should reveal which it
>> is).
>>>> We
>>>>> should also have a relationship mapping from each to the other (e.g.
>>> how
>>>> to
>>>>> convert a signed 32 bit int into a nullable signed 32 bit int.