Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - complex types and ORC


Copy link to this message
-
Re: complex types and ORC
Owen O'Malley 2013-05-03, 21:29
On Mon, Apr 29, 2013 at 4:26 PM, Sean McNamara
<[EMAIL PROTECTED]>wrote:

>  If I create a table that has a map<string, string> field, will ORC files
> columnarize by the keys in the map?  Or will all the pairs in the map be
> grouped together?
>

It will break the map keys into one sub-column and the map values into a
separate sub-column. The map keys will end up in a dictionary and the
values will be in a different dictionary. It would be interesting to have a
map encoding that columnarized low cardinality maps, but I haven't heard
anyone thinking about it.

-- Owen
>  My goal is to columnarize the storage of a variable number of fields
> (where the names and number of fields are unknown at schema creation).  It
> does not appear to be possible with RCFILE, and I'm curious if ORC just
> adds better type support, or if they also columnarize the fields within
> complex types.
>
>  Thanks,
>
>  Sean
>