Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill, mail # dev - Re: Writer interface start


Copy link to this message
-
Re: Writer interface start
Timothy Chen 2013-10-02, 18:43
Hi Jason,

Have some questions around your considerations:

- Maybe a bit more context? A writer interface doesn't seem to suggest what
it really is about. Also if this is focused on writing (from record reader
into drill vv), why is there many comments around reading in your
consideration?
- I don't see any writer interface proposed?
- Some of the considerations you're putting in columnar also applies to row
major as well:
    - compression (ie: Avro compresses per block).
    - schema changes can happen in both
- What are we writing to disk? And why does columnar requires a larger
in-memory structure to be written to disk?
- I don't quite get why row major requires additional objects to be passed
to writer?

Tim

On Wed, Oct 2, 2013 at 11:19 AM, Jason Altekruse
<[EMAIL PROTECTED]>wrote:

> Please provide any feedback on the start of the writer interface described
> in the attached document. It should be a more formalized interface in the
> next few days.
>
> -Jason
>
> ---------- Forwarded message ----------
> From: Jason Altekruse <[EMAIL PROTECTED]>
> Date: Wed, Oct 2, 2013 at 1:12 PM
> Subject: Fwd: Writer interface start
> To: [EMAIL PROTECTED]
>
>
>
>
> ---------- Forwarded message ----------
> From: Jason Altekruse <[EMAIL PROTECTED]>
> Date: Wed, Oct 2, 2013 at 12:31 PM
> Subject: Writer interface start
> To: Jacques Nadeau <[EMAIL PROTECTED]>, Ben Becker <
> [EMAIL PROTECTED]>, Steven Phillips <[EMAIL PROTECTED]>
>
>
> A quick update on the status of the writer interface. I haven't written it
> formally yet but I put together a document describing the important design
> considerations for various formats, trying to be as general as possible.
> Should be more fleshed out in detail in the next few days.
>
> See attached
> -Jason
>
>
>