Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> B[yi]teSize execwork tasks someone could potentially help out with...


Copy link to this message
-
Re: B[yi]teSize execwork tasks someone could potentially help out with...
good point, i'll try and ask the author.
it's a pretty recent lib so that might be an oversight…

-david

On Apr 26, 2013, at 12:04 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:

> Jacques I think this is the one I emailed you before that has no licensing info.
>
> Tim
>
> Sent from my iPhone
>
> On Apr 26, 2013, at 9:30 AM, David Alves <[EMAIL PROTECTED]> wrote:
>
>> i've looked through it and looks like it can leverage shared memory, which I was looking for anyway.
>> I also like the way garbage collection works (gc in java also clears off-heap).
>> I'll take a deeper look during the weekend.
>>
>> -david
>>
>> On Apr 26, 2013, at 11:25 AM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:
>>
>>> I've looked at that in the past and think the idea of using here is very
>>> good.  It seems like ByteBuf is nice as it has things like endianess
>>> capabilities, reference counting and management and Netty direct support.
>>> On the flipside, larray is nice for its large array capabilities and
>>> better input/output interfaces.  The best approach might be to define a new
>>> ByteBuf implementation that leverages LArray.  I'll take a look at this in
>>> a few days if someone else doesn't want to.
>>>
>>> j
>>>
>>> On Fri, Apr 26, 2013 at 8:39 AM, kishore g <[EMAIL PROTECTED]> wrote:
>>>
>>>> Fort *ByteBuf Improvements*, Have you looked at LArrayJ
>>>> https://github.com/xerial/larray. It has those wrappers and I found it
>>>> quite useful. The same person has also written java version for snappy
>>>> compression. Not sure if you guys have plan to add compression, but one of
>>>> the nice things I could do was use the memory offsets for source(compressed
>>>> data) and dest(uncompressed array) and do the decompression off-heap. It
>>>> supports the need for looking up by index and has wrappers for most of the
>>>> primitive data types.
>>>>
>>>> Are you looking at something like this?
>>>>
>>>> thanks,
>>>> Kishore G
>>>>
>>>>
>>>>
>>>> On Fri, Apr 26, 2013 at 7:53 AM, Jacques Nadeau <[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>>> They are on the list but the list is long :)
>>>>>
>>>>> Have a good weekend.
>>>>>
>>>>> On Thu, Apr 25, 2013 at 9:51 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> So if no one picks anything up you will be done with all the work in
>>>> the
>>>>>> next couple of days? :)
>>>>>>
>>>>>> Would like to help out but I'm traveling to la over the weekend.
>>>>>>
>>>>>> I'll sync with you Monday to see how I can help then.
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Apr 25, 2013, at 9:06 PM, Jacques Nadeau <[EMAIL PROTECTED]>
>>>> wrote:
>>>>>>
>>>>>>> I'm working on the execwork stuff and if someone would like to help
>>>>> out,
>>>>>>> here are a couple of things that need doing.  I figured I'd drop them
>>>>>> here
>>>>>>> and see if anyone wants to work on them in the next couple of days.
>>>> If
>>>>>> so,
>>>>>>> let me know otherwise I'll be picking them up soon.
>>>>>>>
>>>>>>> *RPC*
>>>>>>> - RPC Layer Handshakes: Currently, I haven't implemented the
>>>> handshake
>>>>>> that
>>>>>>> should happen in either the User <> Bit or the Bit <> Bit layer.  The
>>>>>> plan
>>>>>>> was to use an additional inserted event handler that removed itself
>>>>> from
>>>>>>> the event pipeline after a successful handshake or disconnected the
>>>>>> channel
>>>>>>> on a failed handshake (with appropriate logging).  The main
>>>> validation
>>>>> at
>>>>>>> this point will be simply confirming that both endpoints are running
>>>> on
>>>>>> the
>>>>>>> same protocol version.   The only other information that is currently
>>>>>>> needed is that that in the Bit <> Bit communication, the client
>>>> should
>>>>>>> inform the server of its DrillEndpoint so that the server can then
>>>> map
>>>>>> that
>>>>>>> for future communication in the other direction.
>>>>>>>
>>>>>>> *DataTypes*
>>>>>>> - General Expansion: Currently, we have a hodgepodge of datatypes
>>>>> within