Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Writing Custom Serdes for Hive


Copy link to this message
-
RE: Writing Custom Serdes for Hive
A serde is actually used the other way around... Hive parses the query, writes MapReduce code to solve the query, and the generated code uses the serde for field access.

Standard way to write a serde is to start from the trunk regex serde, then modify as needed...

http://svn.apache.org/viewvc/hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/RegexSerDe.java?revision=1131106&view=markup

Also, nice article by Roberto Congiu...

http://www.congiu.com/a-json-readwrite-serde-for-hive/

Chuck Connell
Nuance R&D Data Team
Burlington, MA
From: John Omernik [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, October 16, 2012 11:30 AM
To: [EMAIL PROTECTED]
Subject: Writing Custom Serdes for Hive

We have a maybe obvious question about a serde. When a serde in invoked, does it have access to the original hive query?  Ideally the original query could provide the Serde some hints on how to access the data on the backend.

Also, are there any good links/documention on how to write Serdes?  Kinda hard to google on for some reason.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB