Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> update RE: hive map field question


Copy link to this message
-
update RE: hive map field question
Hi Hive users,

I thought I'll update the status on this.  I could spend some time on cracking hive code, map field related in particular,
1.  Explode works great and it solves a very needed problem for me.

2.  There is already a str_to_map udf available in cdh3u2.  I stumbled upon this while I was looking to create a udf

3.  Here comes the awesome part.  I finally created a udf to convert a map to string.  I would like to contribute it to the hive community.  Let me know how I can do that.

4.  Also there are map_keys and map_values functions which would give a list of arrays of keys and values respectively.

Let me know if you have any questions.

Thanks,
Chalcy

From: Chalcy Raja [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, November 23, 2011 1:20 PM
To: [EMAIL PROTECTED]; 'Bejoy Ks'
Subject: RE: hive map field question

Wow, this is great news.  I read about explode and wished map had that.

Thanks so much,
Chalcy

From: Bejoy Ks [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, November 23, 2011 1:05 PM
To: [EMAIL PROTECTED]
Subject: Re: hive map field question

Hi Chalcy
        Support for splitting a column of map data type into key and values is available on the hive latest trunk. You can use a explode() UDF to achieve the same, which used to support only arrays previously.
http://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java?view=markup

You need to patch your hive source installation to get this functionality running. Or as a quick fix (not highly recommended) take the source code include the same as a Custom UDF for your temporary usage. You can utilize the explode functionality as follows combining the same with Lateral Views.

Sample DDL
CREATE TABLE test_map
(
  clmn1 STRING,
  clmn2 STRING,
  clmnMap MAP<STRING,STRING>
)

Sample DML - to extraxt key values from map
SELECT clmn1,clmn2,clmnMap_key,clmnMap_value FROM test_map
LATERAL VIEW explode(clmnMap) cl AS clmnMap_key,clmnMap_value;

Hope it helps!...

Regards
Bejoy.K.S

________________________________
From: Mark Grover <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, November 23, 2011 9:55 PM
Subject: Re: hive map field question

Chalcy,
I don't know of an existing UDF that lists all keys and values of a map. You may want to check ql/src/java/org/apache/hadoop/hive/ql/udf in Hive source to double check.
But, as you might have guessed, it'd be pretty straightforward to do that in your own UDF:-)

Mark

----- Original Message -----
From: "Chalcy Raja" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Sent: Wednesday, November 23, 2011 11:14:09 AM
Subject: RE: hive map field question

Thanks, Mark.  I'll try that.

Also, is there a way to list all the keys and values in a map field?

Thanks,
Chalcy

-----Original Message-----
From: Mark Grover [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, November 23, 2011 11:01 AM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: hive map field question

Hi Chalcy,
You could write a UDF that takes in a Map object and returns you the string you need.

Alternatively, you could use Hive Transform. Details here:https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform

When a map gets passed to your custom transform script, you could read it as a serialized string.

Mark

----- Original Message -----
From: "Chalcy Raja" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Sent: Wednesday, November 23, 2011 10:48:07 AM
Subject: hive map field question
Hello HiveUsers,

I have a need to convert a map<string,string> field to string field and vice versa in a hive table. I could not do cast.

I created two external tables with one has string and another map. I can join both to get what I want, takes a long time.

Any ideas of how it can be done efficiently ?

Thanks,

Chalcy
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB