Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Accessing Table Properies from InputFormat


+
Peter Marron 2013-05-28, 14:59
+
Owen OMalley 2013-05-28, 15:42
Copy link to this message
-
Re: Accessing Table Properies from InputFormat
That does not really make sense. Your breaking the layered approache.
InputFormats read/write data, serdes interpret data based on the table
definition. its like asking "Why can't my input format run assembly code?"
On Tue, May 28, 2013 at 11:42 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:

>
>
>
> On Tue, May 28, 2013 at 7:59 AM, Peter Marron <
> [EMAIL PROTECTED]> wrote:
>
>>  Hi,****
>>
>> ** **
>>
>> Hive 0.10.0 over Hadoop 1.0.4.****
>>
>> ** **
>>
>> Further to my filtering questions of before.****
>>
>> I would like to be able to access the table properties from inside my
>> custom InputFormat.****
>>
>> I’ve done searches and there seem to be some other people who have had a
>> similar problem.****
>>
>> The closest I can see to a solution is to use ****
>>
>>                 MapredWork mrwork >> Utilities.getMapRedWork(configuration);****
>>
>> but this fails for me with the error below.****
>>
>> I’m not truly surprised because I and trying to make sure that my query**
>> **
>>
>> runs without a map/reduce and some of the e-mails suggest that in this
>> case:****
>>
>> ** **
>>
>> “…no mapred job is
>> run, so this trick doesn't work (and instead, the Configuration object
>> can be used, since it's local).”****
>>
>> ** **
>>
>> Any pointers would be very much appreciated.
>>
>
> Yeah, as you discovered, that only works in the MapReduce case and breaks
> on cases like "select count(*)" that don't run in MapReduce.
>
> I haven't tried it, but it looks like the best you can do with the current
> interface is to implement a SerDe which is passed the table properties in
> initialize. In terms of passing it to the InputFormat, I'd try a thread
> local variable. It looks like the getRecordReader is called soon after the
> serde.initialize although I didn't do a very deep search of the code.
>
> -- Owen
>
>
>
+
Owen OMalley 2013-05-28, 16:10
+
Edward Capriolo 2013-05-28, 16:27
+
Owen OMalley 2013-05-28, 17:31
+
Edward Capriolo 2013-05-28, 18:25
+
Peter Marron 2013-05-29, 09:16
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB