Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - CHAN (Comprehensive Hive Archive Network) (Was Re: Using Reflect: A thread for ideas)


+
Robin Morris 2013-02-14, 06:59
+
Connell, Chuck 2013-02-14, 13:58
Copy link to this message
-
Re: CHAN (Comprehensive Hive Archive Network) (Was Re: Using Reflect: A thread for ideas)
Patrick D'Souza 2013-02-14, 16:30
I was having a similar discussion with some of our Hadoop Admins at
About.com last week about a central repository for UD[A,T]Fs. Most often
we're trying to accomplish the same thing with UD[A,T]Fs so a central
repository can make live easy for everyone.
On Thu, Feb 14, 2013 at 8:58 AM, Connell, Chuck <[EMAIL PROTECTED]>wrote:

>  +1, great idea!****
>
> ** **
>
> Chuck Connell, Nuance****
>
> ** **
>
> *From:* Robin Morris [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, February 14, 2013 1:59 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* CHAN (Comprehensive Hive Archive Network) (Was Re: Using
> Reflect: A thread for ideas)****
>
> ** **
>
> I think we need to think a little bigger than this.  ****
>
> ** **
>
> Recently I've been thinking that what would be most useful to the Hive
> user community would be a CHAN – Comprehensive Hive Archive Network,
> (analogous to CPAN, CRAN, CTAN etc.).  A central place where
> user-contributed UD[A,T]Fs could be uploaded and made available to the
> wider community.****
>
> ** **
>
> Is there interest in the user community for something like this?****
>
> ** **
>
> Robin****
>
> ** **
>
> *From: *John Omernik <[EMAIL PROTECTED]>
> *Reply-To: *"[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> *Date: *Wednesday, February 13, 2013 8:38 PM
> *To: *"[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> *Subject: *Using Reflect: A thread for ideas****
>
> ** **
>
> I stumbled across the little documented reflect function today. I've
> always known about it, but java scares me if it's not in a cup so I didn't
> dig.  Well today I dug, and found an awesome use case for reflect (for me)
> and wanted to share.  I also thought it would be nice to validate some
> thoughts I had on reflect, and how we could possibly share ideas on reflect
> so that folks could get more use out of this great feature of hive.  ****
>
> ** **
>
> Here's my example: A simple URL decode function:****
>
> ** **
>
> select url, reflect('java.net.URLDecoder', 'decode', url, 'utf-8') as
> decoded_url from logs****
>
> Basically I am using the decode function of the java.net.URLDecoder class.
>  Pretty awesome, works great, no files to distribute either.  Even works
> through JDBC!****
>
> ** **
>
> Ok that being said, I realized now that the function I am trying to call
> has to return data in a simple data type.  For example, I struggle to come
> up with a simple reflect() for making an Hex MD5 out of a string because
> the built in function return an object, which have methods that can return
> what I am looking for. Which is great, but then I have to compile java
> code, distribute a jar, and then run the code. I am looking for simple like
> the URLDecoding function.  ****
>
> ** **
>
> I love this reflect feature, but I think it's probably underutilized due
> to the perceived usability issues for beginners.  So that leads me to my
> next thought. What if we brain storm here handy functions in Java that are
> not included in the standard hive language, that make the transition to
> hive well using the reflect function and the show an example of it's use? I
> went first with my URLDecode, and obviously will be looking for more, but
> have you seen some examples that we neat and worked well for you? Can you
> share?  ****
>
> ** **
>
> Perhaps if we get enough examples we could roll some of these into a wiki
> page on the hive wiki that folks can use to get over the "perceived"
> complexity of using java reflect?  ****
>
> ** **
>
> Thanks to those who have worked hard to implement features like this, it
> is truly awesome. ****
>
+
John Omernik 2013-02-14, 12:45