Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> B[yi]teSize execwork tasks someone could potentially help out with...


Copy link to this message
-
Re: B[yi]teSize execwork tasks someone could potentially help out with...
I've looked at that in the past and think the idea of using here is very
good.  It seems like ByteBuf is nice as it has things like endianess
capabilities, reference counting and management and Netty direct support.
 On the flipside, larray is nice for its large array capabilities and
better input/output interfaces.  The best approach might be to define a new
ByteBuf implementation that leverages LArray.  I'll take a look at this in
a few days if someone else doesn't want to.

j

On Fri, Apr 26, 2013 at 8:39 AM, kishore g <[EMAIL PROTECTED]> wrote:

> Fort *ByteBuf Improvements*, Have you looked at LArrayJ
> https://github.com/xerial/larray. It has those wrappers and I found it
> quite useful. The same person has also written java version for snappy
> compression. Not sure if you guys have plan to add compression, but one of
> the nice things I could do was use the memory offsets for source(compressed
> data) and dest(uncompressed array) and do the decompression off-heap. It
> supports the need for looking up by index and has wrappers for most of the
> primitive data types.
>
> Are you looking at something like this?
>
> thanks,
> Kishore G
>
>
>
> On Fri, Apr 26, 2013 at 7:53 AM, Jacques Nadeau <[EMAIL PROTECTED]>
> wrote:
>
> > They are on the list but the list is long :)
> >
> > Have a good weekend.
> >
> > On Thu, Apr 25, 2013 at 9:51 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:
> >
> > > So if no one picks anything up you will be done with all the work in
> the
> > > next couple of days? :)
> > >
> > > Would like to help out but I'm traveling to la over the weekend.
> > >
> > > I'll sync with you Monday to see how I can help then.
> > >
> > > Tim
> > >
> > > Sent from my iPhone
> > >
> > > On Apr 25, 2013, at 9:06 PM, Jacques Nadeau <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > I'm working on the execwork stuff and if someone would like to help
> > out,
> > > > here are a couple of things that need doing.  I figured I'd drop them
> > > here
> > > > and see if anyone wants to work on them in the next couple of days.
>  If
> > > so,
> > > > let me know otherwise I'll be picking them up soon.
> > > >
> > > > *RPC*
> > > > - RPC Layer Handshakes: Currently, I haven't implemented the
> handshake
> > > that
> > > > should happen in either the User <> Bit or the Bit <> Bit layer.  The
> > > plan
> > > > was to use an additional inserted event handler that removed itself
> > from
> > > > the event pipeline after a successful handshake or disconnected the
> > > channel
> > > > on a failed handshake (with appropriate logging).  The main
> validation
> > at
> > > > this point will be simply confirming that both endpoints are running
> on
> > > the
> > > > same protocol version.   The only other information that is currently
> > > > needed is that that in the Bit <> Bit communication, the client
> should
> > > > inform the server of its DrillEndpoint so that the server can then
> map
> > > that
> > > > for future communication in the other direction.
> > > >
> > > > *DataTypes*
> > > > - General Expansion: Currently, we have a hodgepodge of datatypes
> > within
> > > > the org.apache.drill.common.expression.types.DataType.  We need to
> > clean
> > > > this up.  There should be types that map to standard sql types.  My
> > > > thinking is that we should actually have separate types for each for
> > > > nullable, non-nullable and repeated (required, optional and repeated
> in
> > > > protobuf vernaciular) since we'll generally operate with those values
> > > > completely differently (and that each type should reveal which it
> is).
> > >  We
> > > > should also have a relationship mapping from each to the other (e.g.
> > how
> > > to
> > > > convert a signed 32 bit int into a nullable signed 32 bit int.
> > > >
> > > > - Map Types: We don't need nullable but we will need different map
> > types:
> > > > inline and fieldwise.  I think these will useful for the execution
> > engine
> > > > and will be leverage depending on the particular needs-- for example