Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Accessing Table Properies from InputFormat


Copy link to this message
-
Re: Accessing Table Properies from InputFormat
Owen O'Malley 2013-05-28, 16:10
On Tue, May 28, 2013 at 8:45 AM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> That does not really make sense. Your breaking the layered approache.
> InputFormats read/write data, serdes interpret data based on the table
> definition. its like asking "Why can't my input format run assembly code?"
>

The current model of:

SerDe
Input/OutputFormat
FileSystem

does well for text formats, but otherwise limits the input/output formats
to doing binary data. That creates problems if the Input/OutputFormat has
an integrated serialization mechanism. For example, ORC requires its SerDe
and the OrcSerde just passes along the values through serialize and
deserialize.

Also note that other formats like SequenceFile are restricted because the
SerDe is placed above the FileFormat. Hive's SequenceFile input format
discards the key and requires the value to be Text or BytesWritable. That
covers many cases, but certainly not all. On the other hand, if it was
Hive's SequenceFile InputFormat that was creating the ObjectInspector, it
could actually handle more complex types and let Hive usefully read a wider
range of SequenceFiles.

I would propose that it would be better to push SerDes down into the
Input/OutputFormats that can be parameterized by the serialization. Using
them for TextInput/OutputFormat and HBaseTableInput/OutputFormat makes a
lot of sense, but in general that isn't true.

-- Owen