Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Better formated.. Pig udf help


+
jamal sasha 2012-10-25, 21:46
+
pablomar 2012-10-25, 22:45
+
jamal sasha 2012-10-26, 17:24
+
Bill Graham 2012-10-25, 23:57
+
jamal sasha 2012-10-26, 17:35
Copy link to this message
-
Re: Better formated.. Pig udf help
Do you see a "conf" dir within /path/Hadoop? If yes, point your
PIG_CLASSPATH to it.

export PIG_CLASSPATH=/path/hadoop/conf.

Sent from my iPhone

On Oct 26, 2012, at 10:35 AM, jamal sasha <[EMAIL PROTECTED]> wrote:

> Hi
> Great catch
> Now I get an error
> Cannot find hadoop configuration in class path ( neither hadoop site XML
> etc)
>
> So I am running the file on a cluster which had say hadoop set up as
>
> /path/hadoop
> /path/pig
>
> And I have account in it
> So I cannot change the hadoop conf files as other users are also using it.
> How do I run this just for me ?
> On Thursday, October 25, 2012, Bill Graham <[EMAIL PROTECTED]> wrote:
>> Somewhere you have a typo, probably in the execution of your program:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/pache/pig/Main****
>>
>> Caused by: java.lang.ClassNotFoundException: org.pache.pig.Main****
>>
>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)****
>>
>> Note that the 'a' in apache is missing.
>>
>>
>> On Thu, Oct 25, 2012 at 2:46 PM, jamal sasha <[EMAIL PROTECTED]>
> wrote:
>>
>>> Hi,****
>>>
>>>  I am trying to write a pig udf function.. Basically the data is of
>>> format*
>>> ***
>>>
>>> ** **
>>>
>>> Id,time****
>>>
>>> What I am trying to do is … parse the time and then see whether its
>>> breakfast, lunch or dinner.. based on the time stamp. Some entries wil be
>>> null as well..****
>>>
>>> ** **
>>>
>>> So here is the udf code for this.****
>>>
>>> ** **
>>>
>>> *public* *class* time *extends* EvalFunc<String>{****
>>>
>>> ** **
>>>
>>>       *public* String exec(Tuple input) *throws* IOException {****
>>>
>>>       ****
>>>
>>>              *if* ((*input* == *null*) || (input.size() == 0))****
>>>
>>>               *return* *null*;****
>>>
>>>           *try*{****
>>>
>>>               String time = (String) input.get(0) ;****
>>>
>>>               DateFormat df = *new*
> SimpleDateFormat("hh:mm:ss.000");****
>>>
>>>               Date date = df.parse(time);****
>>>
>>>               String timeOfDay = *getTimeOfDay*(date);****
>>>
>>>               *return* timeOfDay;****
>>>
>>>           } *catch* (ParseException e) {****
>>>
>>>               //how will I handle when df.parse(time) fails and throws
>>> ParseException?****
>>>
>>>               //maybe:****
>>>
>>>               *return* *null*;****
>>>
>>>           }****
>>>
>>>              ****
>>>
>>>              ****
>>>
>>>       }****
>>>
>>> ** **
>>>
>>> After this.. in eclipse.. I did the export of this as a jar called myudfs
>>> and I have a jar file called myudfs.jar****
>>>
>>> ** **
>>>
>>> Then I wrote the pig script as ****
>>>
>>> Time.pig****
>>>
>>> ** **
>>>
>>> REGISTER path/to/udf/myudfs.jar****
>>>
>>> in = LOAD 'path/to/input' USING PigStorage(',') AS (id:long,
>>> time:chararray);****
>>>
>>> result = foreach in GENERATE  myudfs.time(time);****
>>>
>>> STORE result INTO 'path/to/output' using PigStorage(',');****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> So.. as per the documentation.. http://wiki.apache.org/pig/UDFManual****
>>>
>>> ** **
>>>
>>> Now I did this..****
>>>
>>> java -cp path/to/pig.jar org.apache.pig.Main time.pig****
>>>
>>> ** **
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/pache/pig/Main****
>>>
>>> Caused by: java.lang.ClassNotFoundException: org.pache.pig.Main****
>>>
>>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)****
>>>
>>>        at java.security.AccessController.doPrivileged(Native Method)****
>>>
>>>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)****
>>>
>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)****
>>>
>>>        at
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)****
>>>
>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)****
>>>
>>> Could not find the main class: org.pache.pig.Main.  Program will
> exit.****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> What am I doing wrong?****
+
jamal sasha 2012-11-12, 15:36