|
|
-
Re: Better formated.. Pig udf helpPrashant Kommireddi 2012-10-26, 17:43
Do you see a "conf" dir within /path/Hadoop? If yes, point your
PIG_CLASSPATH to it. export PIG_CLASSPATH=/path/hadoop/conf. Sent from my iPhone On Oct 26, 2012, at 10:35 AM, jamal sasha <[EMAIL PROTECTED]> wrote: > Hi > Great catch > Now I get an error > Cannot find hadoop configuration in class path ( neither hadoop site XML > etc) > > So I am running the file on a cluster which had say hadoop set up as > > /path/hadoop > /path/pig > > And I have account in it > So I cannot change the hadoop conf files as other users are also using it. > How do I run this just for me ? > On Thursday, October 25, 2012, Bill Graham <[EMAIL PROTECTED]> wrote: >> Somewhere you have a typo, probably in the execution of your program: >> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/pache/pig/Main**** >> >> Caused by: java.lang.ClassNotFoundException: org.pache.pig.Main**** >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)**** >> >> Note that the 'a' in apache is missing. >> >> >> On Thu, Oct 25, 2012 at 2:46 PM, jamal sasha <[EMAIL PROTECTED]> > wrote: >> >>> Hi,**** >>> >>> I am trying to write a pig udf function.. Basically the data is of >>> format* >>> *** >>> >>> ** ** >>> >>> Id,time**** >>> >>> What I am trying to do is … parse the time and then see whether its >>> breakfast, lunch or dinner.. based on the time stamp. Some entries wil be >>> null as well..**** >>> >>> ** ** >>> >>> So here is the udf code for this.**** >>> >>> ** ** >>> >>> *public* *class* time *extends* EvalFunc<String>{**** >>> >>> ** ** >>> >>> *public* String exec(Tuple input) *throws* IOException {**** >>> >>> **** >>> >>> *if* ((*input* == *null*) || (input.size() == 0))**** >>> >>> *return* *null*;**** >>> >>> *try*{**** >>> >>> String time = (String) input.get(0) ;**** >>> >>> DateFormat df = *new* > SimpleDateFormat("hh:mm:ss.000");**** >>> >>> Date date = df.parse(time);**** >>> >>> String timeOfDay = *getTimeOfDay*(date);**** >>> >>> *return* timeOfDay;**** >>> >>> } *catch* (ParseException e) {**** >>> >>> //how will I handle when df.parse(time) fails and throws >>> ParseException?**** >>> >>> //maybe:**** >>> >>> *return* *null*;**** >>> >>> }**** >>> >>> **** >>> >>> **** >>> >>> }**** >>> >>> ** ** >>> >>> After this.. in eclipse.. I did the export of this as a jar called myudfs >>> and I have a jar file called myudfs.jar**** >>> >>> ** ** >>> >>> Then I wrote the pig script as **** >>> >>> Time.pig**** >>> >>> ** ** >>> >>> REGISTER path/to/udf/myudfs.jar**** >>> >>> in = LOAD 'path/to/input' USING PigStorage(',') AS (id:long, >>> time:chararray);**** >>> >>> result = foreach in GENERATE myudfs.time(time);**** >>> >>> STORE result INTO 'path/to/output' using PigStorage(',');**** >>> >>> ** ** >>> >>> ** ** >>> >>> So.. as per the documentation.. http://wiki.apache.org/pig/UDFManual**** >>> >>> ** ** >>> >>> Now I did this..**** >>> >>> java -cp path/to/pig.jar org.apache.pig.Main time.pig**** >>> >>> ** ** >>> >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/pache/pig/Main**** >>> >>> Caused by: java.lang.ClassNotFoundException: org.pache.pig.Main**** >>> >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)**** >>> >>> at java.security.AccessController.doPrivileged(Native Method)**** >>> >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)**** >>> >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)**** >>> >>> at >>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)**** >>> >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)**** >>> >>> Could not find the main class: org.pache.pig.Main. Program will > exit.**** >>> >>> ** ** >>> >>> ** ** >>> >>> What am I doing wrong?**** |