Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Create rdbms like sequence in Pig on Pig Relation


Copy link to this message
-
Re: Create rdbms like sequence in Pig on Pig Relation
Are there examples of JRuby UDFs? I couldn't figure it out.

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com

On May 27, 2012, at 4:01 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Right.. Russel, the reason DynamicInvokers weren't working is that
> InvokeForString expects a function that returns a String. randomUUID
> returns a UUID, not a String.
> You could of course call this trivially using jruby udfs (less work
> than the java version).
>
> D
>
> On Sun, May 27, 2012 at 2:39 PM, Dragan Nedeljkovic <[EMAIL PROTECTED]> wrote:
>> You have to call UUID.randomUUID() to get an UUID, but you cannot use DEFINE
>> to do that since DEFINE does not support methods that return arbitrary classes.
>>
>> Wrapping it into an UDF, works just fine,
>>
>> package piggybank;
>>
>> import java.io.IOException;
>> import java.util.UUID;
>>
>> import org.apache.pig.EvalFunc;
>> import org.apache.pig.data.Tuple;
>>
>> public class CreateUUID
>> extends EvalFunc<String>
>> {
>> public String exec(Tuple input)
>> throws IOException
>> {
>> try
>> {
>> return UUID.randomUUID().toString();
>> }
>> catch(Exception e)
>> {
>> // Throwing an exception will cause the task to fail.
>> throw new IOException("Something bad happened!", e);
>> }
>> }
>> }
>> // eof
>>
>>
>> register 'mypiggybank.jar';
>> define CreateUUID piggybank.CreateUUID();
>>
>> input_lines = LOAD 'test_CreateUUID.in' AS (line:chararray);
>> describe input_lines;
>> dump input_lines;
>>
>> new_list = FOREACH input_lines GENERATE line, CreateUUID();
>> describe new_list;
>> dump new_list;
>>
>> -- eof
>>
>>
>>> ________________________________
>>> From: Russell Jurney <[EMAIL PROTECTED]>
>>> To: [EMAIL PROTECTED]
>>> Sent: Sunday, May 27, 2012 4:56:07 PM
>>> Subject: Re: Create rdbms like sequence in Pig on Pig Relation
>>>
>>> It helps, but I am not able to invoke java.util.UUID.toString, maybe
>>> because it doesn't take an argument.  This is from the docs:
>>>
>>> DEFINE UrlDecode InvokeForString('java.net.URLDecoder.decode', 'String
>>> String');
>>> encoded_strings = LOAD 'encoded_strings.txt' as (encoded:chararray);
>>> decoded_strings = FOREACH encoded_strings GENERATE UrlDecode(encoded,
>>> 'UTF-8');
>>>
>>>
>>> Maybe I forgot, but is this how I do it?
>>>
>>> DEFINE UUID InvokeForString('java.util.UUID.toString');
>>> with_uuid = FOREACH my_stuff generate UUID(), *;
>>>
>>>
>>> Sorry, I only understand example code - not APIs. My Java is quite weak.
>>>
>>> http://docs.oracle.com/javase/6/docs/api/java/util/UUID.html#toString()
>>>
>>> On Sun, May 27, 2012 at 2:33 AM, Subir S <[EMAIL PROTECTED]> wrote:
>>>
>>>> I hope this helps. DynamicInvoker feature in Pig. Added in 0.8.0
>>>>
>>>>
>>>> http://squarecog.wordpress.com/2010/08/20/upcoming-features-in-pig-0-8-dynamic-invokers/
>>>>
>>>> Thanks
>>>>
>>>> On 5/24/12, Russell Jurney <[EMAIL PROTECTED]> wrote:
>>>>> Thanks, I mean how do you invoke it directly in grunt> from Pig?
>>>>>
>>>>> I keep messing it up for the last 30 minutes. Should I check the settings
>>>>> on my pacemaker, I feel like Fabio on NyQuil messing with this.
>>>>>
>>>>> On Wed, May 23, 2012 at 10:19 PM, Subir S <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>
>>>>>> Hope this helps ->
>>>>>> http://www.javapractices.com/topic/TopicAction.do?Id=56
>>>>>>
>>>>>> and this ->
>>>>>>
>>>>>>
>>>> http://docs.oracle.com/javase/1.5.0/docs/api/java/util/UUID.html#randomUUID%28%29
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, May 24, 2012 at 10:42 AM, Russell Jurney
>>>>>> <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> How do you invoke java.util.UUID.randomUUID?  There is no invoker that
>>>>>>> doesn't take an arg?
>>>>>>>
>>>>>>> On Sun, May 20, 2012 at 6:26 PM, Rajesh Balamohan <
>>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>>
>>>>>>>> I dont think so. However, its a single line java command. You can
>>>>>> create
>>>>>>>> customUDF for this and use in your code.
>>>>>>>>
>>>>>>>> java.util.UUID.randomUUID();
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB