Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - hive json serde


Copy link to this message
-
RE: hive json serde
Connell, Chuck 2012-09-17, 22:49
A few things that should help...

- There are three ways to handle JSON within a Hive table, get_json_object, json_tuple, and JSON SerDe. These are separate mechanisms. So if you want to use get_json_object, you don't need to load a SerDe.

- With a SerDe, Hive understands JSON fields natively, so you can just use the plain field names, no special function call at all.

- I'm not sure if auxpath accepts a relative path, but I know the full path works.

Chuck

________________________________
From: Mark Golden [[EMAIL PROTECTED]]
Sent: Monday, September 17, 2012 5:58 PM
To: Connell, Chuck
Cc: [EMAIL PROTECTED]
Subject: Re: hive json serde

It works now. Looks like there is a bug in the code.

if you do hive --auxpath ./serde then I get an error but if I get the full path as
hive --auxpath /var/lib/hdfs/serde/ then get_json_object() works.

Thanks for your help. Appreciate it.

CCing hive user group as well.
On Mon, Sep 17, 2012 at 2:54 PM, Mark Golden <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
:-) Thanks. I downloaded and got it compiled as well.

My table has nested JSON values so when I tried to do get_json_object(<column>,'$.<key>') this serde doesn't work in that case. It throws an error. I'll email the developer too.

When you tried did you try this on a cluster of nodes or on a single instance?
On Mon, Sep 17, 2012 at 2:38 PM, Connell, Chuck <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Go to the Downloads link on the project page, near the upper right.
________________________________
From: Mark Golden [[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Monday, September 17, 2012 5:32 PM
To: Connell, Chuck
Subject: Re: hive json serde

Hey Chuck,

Where did you download the prebuild jar from?

On Mon, Sep 17, 2012 at 12:10 PM, Connell, Chuck <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
I used his pre-built jar. No need to compile anything.

Be sure to add “ --auxpath /path/to/jar/dir “ to the Hive command line.

Chuck

From: Connell, Chuck [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Monday, September 17, 2012 3:06 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: hive json serde

I just finished testing this one. No problems found. The developer is also quite responsive to issues raised.  I encouraged him to submit it to the Hive dev team as core code.

https://github.com/rcongiu/Hive-JSON-Serde/

Chuck Connell
Nuance R&D Data Team
Burlington, MA
From: Mark Golden [mailto:[EMAIL PROTECTED]]
Sent: Monday, September 17, 2012 2:56 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: hive json serde

Hi,

I'm looking for a hive json serde that works well. Any suggestions? I did a quick google and found a few but none of them are complete.

Thanks,
Mark.