Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Create rdbms like sequence in Pig on Pig Relation


Copy link to this message
-
Re: Create rdbms like sequence in Pig on Pig Relation
Russell Jurney 2012-05-28, 00:06
Are there examples of JRuby UDFs? I couldn't figure it out.

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com

On May 27, 2012, at 4:01 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Right.. Russel, the reason DynamicInvokers weren't working is that
> InvokeForString expects a function that returns a String. randomUUID
> returns a UUID, not a String.
> You could of course call this trivially using jruby udfs (less work
> than the java version).
>
> D
>
> On Sun, May 27, 2012 at 2:39 PM, Dragan Nedeljkovic <[EMAIL PROTECTED]> wrote:
>> You have to call UUID.randomUUID() to get an UUID, but you cannot use DEFINE
>> to do that since DEFINE does not support methods that return arbitrary classes.
>>
>> Wrapping it into an UDF, works just fine,
>>
>> package piggybank;
>>
>> import java.io.IOException;
>> import java.util.UUID;
>>
>> import org.apache.pig.EvalFunc;
>> import org.apache.pig.data.Tuple;
>>
>> public class CreateUUID
>> extends EvalFunc<String>
>> {
>> public String exec(Tuple input)
>> throws IOException
>> {
>> try
>> {
>> return UUID.randomUUID().toString();
>> }
>> catch(Exception e)
>> {
>> // Throwing an exception will cause the task to fail.
>> throw new IOException("Something bad happened!", e);
>> }
>> }
>> }
>> // eof
>>
>>
>> register 'mypiggybank.jar';
>> define CreateUUID piggybank.CreateUUID();
>>
>> input_lines = LOAD 'test_CreateUUID.in' AS (line:chararray);
>> describe input_lines;
>> dump input_lines;
>>
>> new_list = FOREACH input_lines GENERATE line, CreateUUID();
>> describe new_list;
>> dump new_list;
>>
>> -- eof
>>
>>
>>> ________________________________
>>> From: Russell Jurney <[EMAIL PROTECTED]>
>>> To: [EMAIL PROTECTED]
>>> Sent: Sunday, May 27, 2012 4:56:07 PM
>>> Subject: Re: Create rdbms like sequence in Pig on Pig Relation
>>>
>>> It helps, but I am not able to invoke java.util.UUID.toString, maybe
>>> because it doesn't take an argument.  This is from the docs:
>>>
>>> DEFINE UrlDecode InvokeForString('java.net.URLDecoder.decode', 'String
>>> String');
>>> encoded_strings = LOAD 'encoded_strings.txt' as (encoded:chararray);
>>> decoded_strings = FOREACH encoded_strings GENERATE UrlDecode(encoded,
>>> 'UTF-8');
>>>
>>>
>>> Maybe I forgot, but is this how I do it?
>>>
>>> DEFINE UUID InvokeForString('java.util.UUID.toString');
>>> with_uuid = FOREACH my_stuff generate UUID(), *;
>>>
>>>
>>> Sorry, I only understand example code - not APIs. My Java is quite weak.
>>>
>>> http://docs.oracle.com/javase/6/docs/api/java/util/UUID.html#toString()
>>>
>>> On Sun, May 27, 2012 at 2:33 AM, Subir S <[EMAIL PROTECTED]> wrote:
>>>
>>>> I hope this helps. DynamicInvoker feature in Pig. Added in 0.8.0
>>>>
>>>>
>>>> http://squarecog.wordpress.com/2010/08/20/upcoming-features-in-pig-0-8-dynamic-invokers/
>>>>
>>>> Thanks
>>>>
>>>> On 5/24/12, Russell Jurney <[EMAIL PROTECTED]> wrote:
>>>>> Thanks, I mean how do you invoke it directly in grunt> from Pig?
>>>>>
>>>>> I keep messing it up for the last 30 minutes. Should I check the settings
>>>>> on my pacemaker, I feel like Fabio on NyQuil messing with this.
>>>>>
>>>>> On Wed, May 23, 2012 at 10:19 PM, Subir S <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>
>>>>>> Hope this helps ->
>>>>>> http://www.javapractices.com/topic/TopicAction.do?Id=56
>>>>>>
>>>>>> and this ->
>>>>>>
>>>>>>
>>>> http://docs.oracle.com/javase/1.5.0/docs/api/java/util/UUID.html#randomUUID%28%29
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, May 24, 2012 at 10:42 AM, Russell Jurney
>>>>>> <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> How do you invoke java.util.UUID.randomUUID?  There is no invoker that
>>>>>>> doesn't take an arg?
>>>>>>>
>>>>>>> On Sun, May 20, 2012 at 6:26 PM, Rajesh Balamohan <
>>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>>
>>>>>>>> I dont think so. However, its a single line java command. You can
>>>>>> create
>>>>>>>> customUDF for this and use in your code.
>>>>>>>>
>>>>>>>> java.util.UUID.randomUUID();